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introduction 


Leo Havemann! 
Sarah Sherman? 


Assessment and feedback lies at the heart of the learning experience, and forms a significant part 
of both academic and administrative workload. It remains however the single biggest source of 
student dissatisfaction with the higher education experience. 

(Ferrell, 2012, p.3) 


As technology-enhanced learning (TEL) practitioners, we have noted that practices surrounding 
assessment consume a significant proportion of the time academic staff and students spend interacting 
with learning technologies. Indeed, we would suggest that this is because assessment and feedback are 
increasingly produced and consumed digitally. Here in the UK, post-secondary education-focused 
organisations such as Jisc (2009), the National Union of Students (NUS, 2010 and 2011) and the 
Higher Education Academy (HEA, 2012) have called for increased and improved application of learning 
technologies in assessment practice. The NUS study “Student Perspectives on Technology” was based 
on a range of views from students with differing experiences of, and exposure to, online learning and 
technology. This study recommended that institutions consider ways of making university administration 
more accessible through technology, including online submission of assessment (NUS, 2010). This 
point was echoed in the NUS Charter on Technology in Higher Education, which states that the use of 
technology in institutional administration will simplify and improve processes, including assessment 
and feedback (NUS, 2011). 


This summary of student perspectives appears aligned with many of those at the institutional level, 
which focus on the potential to make administrative and service gains through the application of 
technologies. Yet, moving assessment online has not automatically equated to greater satisfaction 
among students. And furthermore, if the processes of setting assessment tasks, developing work for 
submission, and giving and receiving marks and feedback on this work are understood as core to 
learning and teaching, then the actual and potential digital transformation of such practices must 


surely be considered, first of all, from a pedagogical perspective. 


The HEA’s (2012) report “A Marked Improvement” presents a strong argument for transforming 
assessment methods in higher education, founded on the principle of Assessment for Learning (AfL). 
The authors highlight a “need for institutions to continue to adopt robust technological solutions to 
support assessment and feedback”, going beyond administration and replication of established offline 


practices in online environments. They write: 


Effective use of information systems and learning technologies is a precursor to change in 
assessment policy and practice, efficiencies in staff time and a better experience for students. 
A range of technologies can be employed to systematise and improve the administration of the 


whole assessment cycle from submission of work to assessment boards (involving submission, 


1 Leo Havemann, Learning Technologist, Birkbeck, University of London. 


2 Sarah Sherman, Service Manager, Bloomsbury Learning Environment. 


marking and feedback), and including easy access to student work for external examiners. By 
harnessing relevant technologies, the student experience can be enhanced through better 
access to assessment information, a broader range of tasks, automated or speedier feedback, 
student-student and student-staff dialogue regarding assessment, and support for peer and 


group assessment. (HEA, 2012, p. 16). 


In 2014, the Bloomsbury Learning Environment (BLE)’ Consortium initiated a wide-ranging, two- 
year-long research and dissemination project focusing on the use of technology in assessment and 
feedback. The BLE partners (Birkbeck, London School of Hygiene and Tropical Medicine [LSHTM], 
School of Oriental and African Studies [SOAS], Royal Veterinary College [RVC], University College 
London [UCL] Institute of Education and the University of London) are a diverse set of both small 
and specialist, and larger multidisciplinary institutions, which attract a wide range of undergraduates and 
postgraduates, including 18-21 year olds, international students and mature learners. In this way, we 
believe that the BLE is in some senses a ‘microcosm’ of the wider UK higher education (HE) sector. 
By working collaboratively, the BLE partners benefit from a wide pool of collective knowledge, shared 
expertise and resourcing to ensure they learn from each other and improve practice in TEL. The 
BLE was founded upon technological synergies: for example, all participating members use Moodle as 
their Virtual Learning Environment (VLE) platform. An Advisory Board drawn from the member 
institutions was convened to steer the work of the assessment and feedback project. In the spirit of 
openness, we also welcomed participation from additional institutions including UCL, Loughborough 
University, Oxford Brookes University and the London School of Economics (LSE), as well as 


organisations including Jisc and the Association for Learning Technology (ALT). 


The main purpose of the project was to understand and improve online assessment and feedback 
processes, practices, opportunities and technologies available to the members of the BLE Consortium. 


There were three main aims and objectives: 


1 to identify various stakeholders and their expectations and goals in the deployment and use of 


technology to support assessment and feedback; 


2 to assess and evaluate appropriate emerging technologies which would potentially support 


and enhance assessment and feedback across the consortium; 


3. to produce documentation and case studies, and organise events in order for academic and 


support staff to learn and contribute. 


As we began to produce and collect overviews of current practice and case studies of both technology- 
enabled pedagogy and technical development, we realised that these were of potential interest to a wider 
audience beyond the BLE membership and the idea of collecting them together as an open resource 


began to form. This book therefore represents the culmination of our third project objective. 


3. www.ble.ac.uk 


We have organised the book into a series of sections. The first section contains three papers, which 
attempt to capture macro-level snapshots of current practice (at the time of writing); here, we focus on 
technology use across the assessment lifecycle, the roles played by administrative staff in assessment 
processes, and technology-supported assessment in distance learning, respectively. The four subsequent 
sections contain case studies of digital assessment and feedback practices, which operate at the micro- 
level of specific modules to give insight into the pedagogy underlying the adoption of particular tools, 
and the associated benefits and challenges. The final section contains case studies of technical developments 
which have been undertaken locally to support or enhance aspects of practice. As such, the book 
provides a flavour of the variety and breadth of the BLE’s activities relating to the project theme. The 
issues raised in these chapters remain as vital now as they were when the project theme was first envisaged, 
and although the project itself must end, the work must be ongoing. We are therefore pleased to offer 
this compendium as a contribution to the sector’s widening conversation about the interplay of 


assessment, feedback, pedagogy and technology. 
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SECTION 1 


Contexts 


Chapter 1: Contextualising the Electronic Management 
of Assessment Lifecycle in Bloomsbury 


Leo Havemann‘ 
Sarah Sherman® 


Introduction 


“Teachers have traditionally used assessment to act as both goal and feedback for the learner” 
(Laurillard, 2001, p.58). Assessment and feedback are core to student learning, and ‘the student experience’ 
of higher education. In recent years, we have seen an increasing interest across the sector in Assessment 
for Learning (AfL), rather than simply of learning (HEA, 2012). In addition, assessment is increasingly 
being submitted, being marked or entirely taking place online, usually because of the affordances of 


technology in managing processes more effectively and efficiently. 


In recent years, UK agencies such as Jisc, the NUS and the HEA have dedicated significant research and 
project resources into the enhancement of assessment and feedback. The NUS has argued that institutions 
must consider ways of making university administration more efficient through technology including 
online submission of assessment (NUS, 2010), and that the use of technology in institutional administration 
simplifies and improves processes, including assessment and feedback (NUS, 2011). Beyond these kinds 
of administrative gains, there is evidence that learning technologies are playing a significant role in the 
transformation of assessment pedagogy. For Jisc (2010), a holistic approach which considers the 


technical, administrative and pedagogic elements of assessment and feedback is necessary. 


If it is now understood that well-designed assessment and timely, forward-focused feedback are essential 
components of a desirable ‘student experience’, then enhancement of assessment and feedback practices 
must be regarded as a priority. Within the institutions that make up the Bloomsbury Learning Environment 
(BLE) Consortium, there has been increasing interest in this area. Sector-wide, institutions have been 
grappling with low scores for feedback in the National Student Survey: “students express concerns 
about the reliability of assessment criteria, challenge the fairness of their experience and say they are 
dissatisfied with the nature and timing of feedback” (HEA, 2012, p. 10). 


To address this locally, the BLE Consortium launched a project in September 2014 to investigate online 
assessment and feedback processes, practices, opportunities and technologies available to the partner 
members known as the ‘Bloomsbury Colleges’ (Birkbeck, UCL Institute of Education, LSHTM, SOAS 
and RVC). The purpose of the Bloomsbury Enhancing Assessment and Feedback Project was, first of all, 
to map current ‘typical’ practice from across the consortium in the use of learning technologies in 
assessment and feedback; and, furthermore, to discover and disseminate innovative approaches via events 
and case studies that showcase how technology can improve assessment. There are good examples of 
widespread adoption of technologies, and also of innovative assessment design which have already 
been developed within the BLE, and our intention was to surface this work in order to encourage and 


inform replication and modification. 


4 Leo Havemann, Learning Technologist, Birkbeck, University of London. 


5 Sarah Sherman, Service Manager, Bloomsbury Learning Environment. 


In this paper, we report on the first strand of the project in which we set out to gain an overview of 
current practice. We have been fortunate that our concerns in this project have significantly overlapped 
with those of the Jisc Electronic Management of Assessment (EMA) programme, and, consequently, 
we have been able to benefit from the work Jisc has produced on behalf of the sector. In order to map 
our data on EMA practices against a relevant framework, we adopted Jisc’s (2016) assessment lifecycle, 
originally developed by Manchester Metropolitan University (Forsyth, Cullen, Ringan and Stubbs, 2015); 
see Figure 1. The lifecycle is an end-to-end model of the stages of the assessment and feedback process, 
which enables consideration of distinct areas of assessment activity and the associated technologies. 
Sub-processes include: assessment scheduling; submission of assignments; tracking of submissions; 
extension requests and approvals; academic integrity; academic misconduct processes; examinations; 
marks recording; moderation and external examining (Jisc, 2016). Using this framework, we have been 
able to categorise practices to demonstrate the various ways in which technology impacts, supports and 


enhances the various stages from setting and submission through to marking and providing feedback. 


The assessment and feedback lifecycle 
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Based on an original idea developed by Manchester Metropolitan University 


Figure 1: The assessment and feedback lifecycle (Jisc, 2015) 


Approach 


Data collection 


There had not previously been any research carried out across the Colleges of the BLE Consortium 
which attempted to collate the various technological approaches to managing assessment and providing 
feedback, either already in use or under consideration. Anecdotal evidence tends to point to both 
diversity and complexity in assessment and feedback practices (often even within a single institution), 
and it is difficult to be sure exactly how diverse or complex such practices are. Therefore, we developed 
a matrix, which was shared with staff responsible for supporting learning technologies within the 


Colleges for completion. 
Analysis 


As EMA practices continuously evolve, this paper aims to explore a snapshot, captured at the time of data 
collection, across the campus-based and blended programmes within the BLE Colleges. As discussed 
above, a key element of the Jisc EMA work has been the framing of assessment and feedback practices 
in terms of the assessment lifecycle. We have structured the discussion of the findings around the 
elements of the lifecycle, about which there is more to say on some than others. When collecting the 
data, we did not start with the intention to research the assessment lifecycle itself, but rather to simply 
map current assessment practices to the technologies used to support them. However, once we had 
collected the data, we concluded that the lifecycle provided a productive analytical framework. The 


following sections describe each stage and corresponding examples from the Colleges. 


Findings 


Specifying 


Specifying is the process of determining the details of a course or programme of study and consequently 
the assessment strategy within it (Jisc, 2016). While the fields in our matrix did not really address the 
specifying stage in the lifecycle, our project as a whole aims to influence assessment strategies. An aspect 
of the project was to capture and highlight innovative practices in technology-enhanced assessment, in 
order to share examples and advice with academics and ultimately support the assessment design process. 
The HEA (2012) concurs with Jisc in highlighting the importance of a programme-level approach to 
specifying assessment. A key challenge here is that assessment design is usually agreed a long time in 


advance, at the time of course/module approval. 
Setting 


Setting assignment details needs to happen each time a group of students takes a particular module; 
this is often known as an instance of delivery. Early on in the lifecycle of an assessment, a decision point 
must establish the key features and format of the assignment, e.g. formative or summative; online or 
offline; coursework or exam; weighting; and timing. It is therefore an outcome of the setting process 
that students receive details, usually in the form of an assignment brief, about precise topics, deadlines, 


learning outcomes assessed, marking criteria and feedback arrangements (Jisc, 2016). From the 


perspective of this project, we were particularly interested in the processes involved in deciding 


whether to conduct assessment online or offline. 


It is not always possible or appropriate to complete assessments online. A good example of an assessment 
that is not suitable for online submission is the Objective Structured Clinical Examination (OSCE), 
which assesses the practical skills of final year veterinary students; similarly, we are not tending to see 
performances, artwork or laboratory-based experiments assessed online. Transposing a written exam 
into an online environment may be considered too high risk or complex. Concerns around students 
proving their identity are commonly expressed in relation to online examinations in particular, 


although this unresolved tension exists in offline assessment too. 


Examples of assessment types that are supported by technological approaches include quizzes, such as 
Multiple Choice Questions (MCQs), which are set up within Moodle and are used frequently by Birkbeck, 
LSHTM and the RVC. The availability of instant, tailored feedback provided to the student is very 
powerful, and so are the facts that MCQs can be accessed any time, any place and used in private, 
multiple times. At the RVC, students are creating their own ‘flashcards’ for self-study, which lends an 
additional learning dimension. The benefits of these types of assessment have been examined informally 
by the Colleges that use them. The ability to provide instantaneous, automatic marks and produce 
statistical analyses are all great benefits. However, a major drawback is that it can be time-consuming 
to create good questions (although once the time has been invested, they can be re-used). Additionally, it 


is hard to assess higher-order skills when deploying these types of formative assessment. 


Across the Colleges, there is little evidence of the use of online summative assessment (e.g. online exams). 
SOAS has some experience of running practice essays conducted online, but it is not very widespread. 
Similarly, the RVC’s Continuing Professional Development Unit has experience of managing open book 
MCQ quizzes in Moodle (randomised questions, categorised by topic and level of difficulty, drawn from 
a wider repository), but this is not mainstream for the whole College. A growing number of suppliers 
of online examinations now exist, but a lack of resourcing and strategic drive have been obstacles 
within the Colleges to investigate the options. The Bloomsbury Enhancing Assessment and Feedback 


Project provided the opportunity to organise collective demonstrations of the options available. 


Supporting 


This component looks specifically at supporting students during the period between Setting and 
Submitting assignments, i.e. while they are in the process of completing an assignment; it has a 
relationship with the broader information and digital literacies agenda for both staff and students 
(Jisc, 2016). Supporting students’ ability to understand the language of assessment and assessment 
processes (known as assessment literacy) has been described by the HEA to be fundamental in helping 
them to appreciate “principles of sound assessment, the relationship between assessment and learning, 
and the nature of professional judgement”. Therefore, it is recommended that students be clearly 
informed about the assessment ‘safeguards’, such as second marking, moderation and external 
examining (HEA, 2012, p. 14). 


The originality reports generated by Turnitin’ are used by some Colleges as formative exercises for students 
to develop their writing abilities while drafting their assignments reports; students are therefore 
encouraged to self-check. Some academics choose to allow students to see the results of the reports, 
allowing for resubmissions and/or late submissions, while others do not. There is a sentiment at the 
UCL Institute of Education in particular that Turnitin should be used to help to empower students 
rather than to identify students who are plagiarising. At the RVC, students are encouraged by their tutors 
to examine their own originality reports, which are made available to them. Several ‘draft’ submission 
areas are provided where students are encouraged to submit draft copies of their work prior to the 
final version. At SOAS and LSHTM, guidance is available to students to help them understand how to 


use Turnitin and how to interpret the originality report. 


Here, the difference between formative and summative assessment types is important: formative 
assessments are primarily about supporting students to learn and ultimately to perform better in later 
summative assessments. Understanding where rapid formative assessments fit into the lifecycle has been a 
challenge, as these arguably encapsulate the entire lifecycle, including feedback. Personal Response Systems 
are used by several of the Colleges to collect student’s responses to formative assessments live in 
classrooms. These systems are useful as they provide support to students in testing their knowledge and, 
therefore, preparing for future, high-stakes assessments. However, their use has been limited since 
academics often have to organise and set up the technology themselves. At the RVC, TurningPoint is 
supported institutionally, but some academics have experienced technical problems, which put them off 
trying again; conversely, a few successful academics have persisted and continue to use it very effectively. 
At the time of writing, LSHTM staff had been given access to PollEverywhere, and were testing it out 
to gauge student understanding and gather feedback. 


In terms of assessment practices which support students’ assessment literacy, we are seeing growth in 
Bloomsbury in the use of peer assessment and feedback. Peer assessment enables a deeper understanding 
of a topic, encourages engagement in group work and supports and improves the student’s assessment 
literacy, allowing them to grasp what is expected of them. In terms of providing opportunities for peer 
assessment, where the students mark each other’s assessment, the UCL Institute for Education has examples 
where this is built into some formative activity designs. At Birkbeck, Criminology students give and receive 
peer feedback on drafts as part of an academic blogging task (see the case study by Guetcherian, Chapter 
14). At the RVC, the Moodle Workshop Tool and WebPA application have been used for peer assessment 
(see case studies by Sherman and Channon, Chapter 10, and Croall, Cobb, Lawson and Spark, Chapter 11). 
In general practice in Bloomsbury, however, peer assessment techniques are not widely used; one 


argument has been the assumption that students don't appreciate or trust their peers’ feedback. 


Submitting 


Submitting is the process of students handing over their completed assignment so that marking and/or 
feedback can take place (Jisc, 2016). This appears to be the area of assessment in which the use of technology 

has become the norm rather than the exception (although its use is by no means universal). In this stage, 
we examined the use of online submission for written coursework as formative and summative assessment. 


The native Moodle and integrated Turnitin assignment tools are used across the Bloomsbury Colleges 


6 Turnitin is a commercial platform for assignment submission and ‘similarity checking’ (in order to detect or deter 
plagiarism), which also offers online marking and feedback functionality. 


(to varying, mixed degrees) to enable electronic submission. The vast majority of Birkbeck coursework 
and all essays at SOAS are submitted via Turnitin. At SOAS, students can submit as many times as they 
wish before the assignment due date in order to view the originality report and self-check their work 
(see ‘Supporting’). While electronic submission can certainly bring advantages in terms of saving students 
and administrators time in the production, delivery and processing of hard copies, it tends to create a 
significant support workload for IT helpdesks and learning technology teams, and of course 


administrators, who experience new demands on their time. 


At the time of writing, the Coursework plugin for Moodle was being developed and trialled at the RVC 
whereby assignments were submitted via Moodle and marked online (see Chapter 27 by Sherman and 
Spark). Turnitin is used extensively at the College in summative assessment for plagiarism detection only; 
this has become mandatory policy. Two high-stakes research projects that are submitted by students 
enrolled on the largest course at the RVC (the veterinary medicine undergraduate course) rely heavily 
upon Turnitin, and the Coursework plugin. A dedicated teaching session before this submission is due 
takes place to assist students with their academic writing skills and how to use Turnitin. At the time of 
data collection for this review, paper copies at the RVC were still required in some instances. LSHTM 
had completed pilots of submission of formative essays via the Assignment tool, but intended to scale 


up online submission using the Coursework plugin. 


Although Moodle and Turnitin support submission of various file formats, the majority of submissions 
are documents. Exceptions to this exist in subject areas where the assignment is to produce a certain 
type of file or dataset (for example, computer science, where a submission might consist of code). 
A growing subset of formats which is less subject-specific is sound and video based assignments: at 
Birkbeck, this has been occasional, but is starting to increase (see the case study by Grange and Hein, 
Chapter 4); at the UCL Institute of Education, some specialist modules require the submission of self- 
made films, animations and audio recordings; at SOAS, some language courses require students to 
submit podcasts for formative assessment. Our existing learning and assessment technologies provide 
opportunities to explore these options more widely, but training and support for both teaching staff 


and students are required. 


Marking and production of feedback 


This crucial stage describes the ways in which submitted work is marked by tutors against evaluation 
criteria and the ways feedback is produced. While marking and the provision of feedback are often 
performed at the same time, they can be understood as distinct functions. It is often in this aspect of the 
lifecycle that we see variation across institutional, local and perhaps disciplinary practices; for example, 


in the application of anonymity and moderation or double marking. 


Turnitin’s GradeMark feature is used extensively at Birkbeck and SOAS in particular, for managing the 
marking process and providing written feedback. At the RVC, GradeMark is used to mark formative 
assessment and the Coursework plugin is used when assignments are required to be annotated online 
and double-blind marked. Alternatively, at the UCL Institute of Education, comments for summative 
assessment have often been produced on a Microsoft Word pro-forma document. However, more 
recently, feedback is required to be returned via the Moodle Assignment plugin, although use of email 
for returning feedback is still widespread. LSHTM ran a pilot of feedback delivery via the Moodle 


Assignment tool, also requiring the use of a Microsoft Word pro-forma. 
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How marking is actually carried out by teaching staff varies between the Colleges. For example, online 
marking in Turnitin GradeMark (including via the iPad app) is quite common at Birkbeck; some tutors 
prefer to download the assignments in order to use the track changes feature in MS Word. On the other 
side of the spectrum, paper-based marking still prevails at the UCL Institute of Education; offline but on- 
screen marking on desktop computers and laptops is, however, on the increase. Again, online marking 


is limited to GradeMark. At SOAS, all assignments which are submitted via Turnitin are marked online. 


Across the Colleges, there has thus far been limited use of audio and video feedback. The occasional 


use of the Campus Pack podcasting tool in Moodle has been made at SOAS in language classes. 
Recording grades 


Once marking has taken place - often by more than one marker -— the grades have to be collated and a 
final mark decided upon. As Jisc (2016) notes, “Institutional regulations will determine who records 
the grade, how this is verified and in which system it is stored. However, in most cases, the student 
record system is the definitive source of grading information” (p.17). Integration between Moodle/ 
Turnitin GradeMark and the institutional student record system has been most formally developed at 
SOAS (see Chapter 25 by O’Sullivan and Leedham). There, grades added to GradeMark within Moodle 
are passed in bulk, on demand, into UNITe (SOAS’s student record system) using a locally developed 
plugin. At the UCL Institute of Education, a similar integration project is also underway. An automated 
update of student records from Moodle and/or Turnitin is not seen as desirable in these cases as student 
records are regarded as the official and definitive repository of final grades, as determined by exam boards. 
At the RVC, integration between the systems is not currently required; the manual input of marks is 


performed by the Exams Office. 


Returning marks and feedback 


This stage is the most important from the student’s point of view, as it describes the way in which they 
receive their marks and feedback. In Bloomsbury, again, there is a widespread reliance on Turnitin as 
the location for marks and feedback to be stored and provided to students. Although Turnitin allows 
for audio feedback to be given, there has been very low uptake of this option. A very low number of 
students will be presented with audio-visual files containing their feedback or via a podcast feed (in 
the case of SOAS’s Language Centre). The availability of alternative, digital forms of feedback (for 
example, audio feedback for language students) might increase levels of satisfaction and increase 
student engagement with it. Constraints of time, resourcing and support requirements appear to be 


barriers to adoption. 


Reflecting 


The final stage of the lifecycle is said to be where the real learning takes place as students consider 
their marks and feedback in the context of the learning objectives in order to reflect on their 
progression. In fact, Jisc argues that it is vital that students “engage with their feedback and use it to 
improve their future performance” (Jisc, 2016, p.20). Teaching staff also use this time to reflect on how 


they can reshape the curriculum or teaching content to affect future results. 
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A good example illustrating the Reflecting stage comes from the UCL Institute of Education (UCL IOE) 
which, as part of its Jisc-funded Assessment Careers project,’ developed a feature in Moodle to allow 
teaching staff to access all the comments and feedback made to individual students of all their assessed 
work. This allows a tutor to identify any commonalities or concerns about individual students and 
enables appropriate interventions to be made. Building upon this, UCL subsequently enhanced the 
feature by enabling the students themselves to view all their feedback in one place (see Chapter 26 by 
Gramp). This enhancement allows a greater opportunity for students to actively take account of the 


feedback they get on an ongoing basis. 


The UCL IOE also explored the potential of using two feedback cover sheets under two key objectives: 
firstly, to improve the relevance of feedback provided to students; secondly, to encourage students to 
make use of all the feedback they receive from staff with a feed-forward, ‘action on feedback’ component, 
which asks students how they addressed any feedback on an earlier version. The feedback cover sheets 
allow for a dialogue between staff and their tutors to open up. Students receiving detailed written 
feedback on complete drafts of all work is very specific to UCL IOE practice rather than a sector norm, 
but the two feedback cover sheets can be used in any case where students receive formative feedback 
before a summative assignment. For the first formative assignment, students completed a feedback 


cover sheet self-evaluation at the time of submission to collect their reflections on the assessment. 


At the formative stage, the feedback cover sheets asked students to complete the following statements: 


— What have you done well in this assignment? 
— What could be improved? 
— Is there anything you specifically want feedback on? 


At the summative stage, the Student Feedback Response Form asked students: 


— Thinking about the feedback on your draft (previous) assignment, please indicate what the 
key points were. For each point state what action you took to respond to this feedback in 
preparing the final version of your assignment. Your response will help your assessor 


identify the progress you have made and suggest further action to help you develop. 


Initially, staff were concerned that the feedback cover sheets might cause additional workload. In 
terms of using the feedback cover sheet for formative work, the IOE concluded that a certain amount 
of ‘assessment literacy’ was required to use them, with the higher achieving students more able to self- 
assess, but staff concerns were allayed and they valued the insights from students’ self-assessment. In 
terms of the cover sheet for summative work, staff found student accounts of what they had done as a 


result of feedback a very helpful starting point for addressing their feedback. 


7 See: For more information see: Assessment Careers Project (IOE/JISC) (video). Available: 
https://www.youtube.com/watch?v=sx_j2VPMiE0. 
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Conclusions 


Considered planning of assessment practices benefits both students, staff and an institution as a whole. 
Exposure to a variety of assessment approaches and formats reinforces learning and plays to learners’ 
different strengths, providing experiential and authentic assessments (see the case study by Sherman 
and Rofe, Chapter 13). For staff, exploiting technology for the setting and marking of an assessment 
can, in the long term, reduce workload. In the case of the institution, integrating platforms, such as 
students record systems and Moodle-based assessment tools (as can be seen at SOAS), is very valuable 


in terms of reducing staff time spent on manual data-entry and the associated risk of errors. 


The lifecycle used in this paper provided a very helpful structure to define and describe the various 
assessment elements and sequences. Much of the lifecycle can potentially be managed and supported 
by deploying electronic systems, but practice across Bloomsbury is quite variable. The key stages that 
are enhanced by the use of technology are Submission, Marking and Production of Feedback. The 
analysis of our data in the context of the lifecycle has provided a useful starting point for investigating 
EMA in Bloomsbury, but more work could usefully be undertaken across all the stages. We initially 
used the lifecycle as a structure to frame data we had already collected rather than to ask questions 


specifically appropriate for the lifecycle. 


The lifecycle framework has helped to identify ways in which the Colleges can use technological 
approaches to manage assessment practices to the benefit of all key stakeholders. Therefore, it is 
recommended that course directors and designers are encouraged to consider, review and challenge 
their own assessment methodologies and the way they design assessments. The HEA (2012) paper on 
assessment remarks on the importance of assessment design for influencing the quality and amount of 
learning achieved by students. It concludes that assessors need to consider a range of assessment methods 
to “improve their validity, authenticity and inclusivity, making them clearly relevant and worthwhile 
in the eyes of students” (HEA, 2012, p.17). 


The examples provided in this paper reveal a high level of dependency on reliable, stable technology, 
in particular on Turnitin, which is a third party, commercial platform. The high-stakes nature of this 
dependence poses a risk (e.g. technical failures or commercial collapse) as well as offering an opportunity 
as this integration provides flexibility and additional functionality. For less high-stakes assessment, 
Personal Response Systems can be used to effectively represent a complete ‘mini-cycle’ comprising the 


Setting, Submitting and Feedback stages in a rapid way. 


The decisions made at the crucial Setting stage, regarding whether or to what extent an assessment 
will be managed online, does not appear to be made on a consistent basis across - or even within - the 
Colleges; there is clearly still potential for the development of policy and guidelines regarding good 
practice in this area. However, the collaborative approach of the Bloomsbury Enhancing Assessment 
and Feedback Project has provided a rich and wide pool of experience from which to extract working 
examples of using online assessment methods. The BLE Consortium has a long history of sharing good 
practice between colleagues across the membership base, and it is hoped that further work in the field 


of EMA in Bloomsbury will continue in a similar, joined-up spirit. 
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Chapter 2: Electronic Management of Assessment — 
Administrative Perspectives 


Leo Havemanné 
Sarah Sherman?’ 


Introduction 


Assessment and feedback are generally understood as key elements in the work of academic staff and in the 
education of students; this realm of activity also often forms a major proportion of the workload of the 
administrative staff who support programme delivery. The roles performed by administrators can tend 


to be overlooked in discussions around the management of assessment methodologies and practices. 


In May 2015, the Bloomsbury Learning Environment (BLE) Consortium organised an event specifically 
for administrative staff from the BLE partner institutions, who were invited to come together and share 
their practice in the use of technology to support and manage their roles. One of the main aims of the 
afternoon was to discuss how attendees were involved in the process of the electronic management of 
assessment (EMA), to inform the BLE project investigating online assessment and feedback processes 


and practices. 


There were a total of 115 registrations for the event; out of these, 77 administrative staff from Birkbeck, 
LSE, LSHTM, Oxford Brookes University, RVC, SOAS and UCL attended (although not all were 
‘participants’ in the EMA exercise). Attendees were asked to create ‘process maps’ depicting the EMA 
workflow in their situation and indicating where their input was required. In total, 20 process maps 
were generated: 13 in which the process being described was (in our view) reasonably clear and detailed, 
and 7 in which we found it to be somewhat unclear. All the maps also contained additional observations 


including the explicit mention of ‘pain points’. 


This paper serves to capture the ways in which administrators are involved in EMA activities and 
highlights the challenges it poses to them. It is important to consider that the information referenced 


here was correct at the time of the event and represents the views of the participants taken on that day. 


Approach 


The role of ‘administrator’ is used in a broad sense for the purpose of this paper; event attendees 
identified themselves as being academic, course, programme, faculty, departmental, research or 
project administrators, as well as examination officers, team leaders and school or course managers. A 
small number of attendees commented that they are not closely involved with EMA processes. As we 
suspected this might be the case, and that some attendees would be colleagues who work together, we 
suggested they should work together in small groups to map out the various activities involved in 


setting up and managing the assessment process from their own context. We provided A3 paper and 


8 Leo Havemann, Learning Technologist, Birkbeck, University of London. 


9 Sarah Sherman, Service Manager, Bloomsbury Learning Environment. 
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multi-coloured felt-tip pens and asked attendees to visually represent how the following elements of 


EMA mapped to each other, indicating their individual involvement: 
e Pre-submission (e.g. setting, setup, anonymity) 
e Submission (e.g. online, paper, dual) 
e Pre-marking 
e Marking (first, second markers, moderation) 
e Feedback/return to students 
e Post feedback (e.g. external/exam board) 
Attendees were also asked to consider and discuss the following questions: 
e Howis EMA managed? What is your involvement? 
e How are markers/moderators assigned? 
e What works well? 
e What doesn’t work so well (pain points)? 


e How might it be improved? 


Below and overleaf are three examples of the process maps developed at the event: 
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Findings 
Processes in the EMA workflow 


Across the sample of administrators that were consulted in this study, a number referred to common 
practices within the process of managing online assessment, but there was also variation. Regardless, it 
should be noted that these experiences alone cannot be used to draw assumptions around general 
practice across an entire institution, as it appears there is not a standard approach to EMA taken by 
any one of the BLE partners. The similarities and differences of the experiences are described in this 
section, and are based on the four administrative-focused aspects of the Jisc (2016) assessment life-cycle 


(Submitting; Marking and production of feedback; Recording grades; Returning marks and feedback). 


Submitting — Before online submission can take place, procedures need to be put in place, such as 
guidelines for teaching staff, dates and deadlines. The format of the assignment activity has to be agreed; 
for example, whether it is to be submitted via Turnitin or the Moodle Assignment tool. Finally, a 
decision around whether the markers receive the papers anonymously has to be made. When online 
submissions are made in Bloomsbury, they may also include a hard copy submission, which the 
administrators themselves may be tasked to print. In addition, in some cases, two paper copies are 
required. Some departments provide signed or emailed receipts of submissions. Extensions might be 


managed by email exchange between students and staff. 


Marking and production of feedback — The administrative staff reported that the allocation of 
assignments to markers is usually made by the lead academic, but in rare cases the administrator is 
responsible for recruiting both first and second markers. In general, the BLE partners differ - both 


internally and from each other - in the way markers are allocated and whether rubrics are used. 


Administrators are often involved in the physical allocation of the hard copy assignments and/or 
downloading and providing markers with electronic copies, except where academics mark online. In 
some cases, administrators manage the process of anonymous and/or double marking, which can 


involve online submission, but is otherwise handled offline. 


There are many extra tasks that a few administrators are expected to manage, which would more usually 
be the responsibility of an academic. For example, checking the Turnitin similarity reports, checking 
students have submitted and then chasing students who have failed to submit and setting marking 
deadlines (which includes informing academics when they should start marking). In some cases, students 
are expected to inform administrators directly when they have submitted. In preparing the marking, 
some administrators have to provide hard copies of marking grids/sheets or upload marking rubrics 
(if used). Administrators may then be expected to enter the assignment and submission data into the 


relevant systems to record the information. 


Across the consortium, it also varies over who publishes and informs students that their results are 


available, for example, the administrators or the academics themselves. 


Recording grades — In many cases, administrators coordinate the liaison between first and second 
markers, which usually involves sharing spreadsheets containing marks and communicating via email 
or at face-to-face meetings. Although the final agreed marks and feedback are accessed by students in 
Moodle, they may be entered either by academics or administrators; all mark sheets are, generally, 


returned to administrators. 
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Not all partners have integrated their student record system with the Moodle grades database. This is a 
big piece of work to develop, but the payoff is very high in terms of saving time and improving 


accuracy and efficiency. 


Returning marks and feedback — Administrators tend to enter marks on their institution’s student 
record system, which can either be a manual process, the upload of a CSV file or by ‘push of a button’, 
which activates an automated transfer from a holding table into the system. This is already taking 


place at SOAS, as the case study in Chapter 25 by O’Sullivan and Leedham describes. 


In terms of moderation, often only a sample is sent to external markers; this may be in hard copy, but 


sometimes they are given special access to the relevant Moodle courses. 


One example that stood out as an interesting case was where an academic completed a feedback form, 
which was scanned by the administrator, saved online and stored. The marks were entered into 


Moodle and the student record system, but the student was provided with a hard copy. 
Challenges 


The administrators highlighted areas they found particularly challenging, which have been sorted into 


the following seven categories. 


Time — Administrators felt that they spent too much time working on extraneous processes that could be 
better used to support students. For example, the duplication of managing paper and online submissions 
is avoidable if the processes are fully digitised. Requiring students to physically travel to hand in an 
assignment was deemed unfair and unnecessary. Administrators also spent time chasing academics to 


receive their marks; and when deadlines were not met, it was the students who inevitably suffered. 


Training — Administrators complained about the lack of available training on the systems they use to 
manage the assessment processes, for themselves, academic staff and for students. Turnitin was the 
most frequently mentioned technology that required support - for example, students’ interpretation 
of the similarity index score. Pedagogic training to enable academics to produce consistent feedback 


was also recommended. 


Technological barriers — Comments regarding unreliability and flaws with the Moodle and Turnitin 
Assignment plugins, which are both used widely to support online assessment, related to the issues with 
handling the anonymity of assignments and the difficulties in enabling first and second (and even 
sometimes third) markers to comment, mark and agree grades. It was therefore felt that Turnitin does 
not completely meet the needs of the UK HE assessment practices, thus explaining why academics still 


mark in hard copy. 


Administrators made reference to the use of ‘too many’ platforms and systems that are not integrated 
(e.g. student record systems, Turnitin and Moodle grades). One participant expressed a worry that, due 
to the manual nature of the procedures and lack of integration of systems, a single point of failure in 
the workflow would be the individual administrator. Incompatible file types and large file sizes (in the 
case of videos for example) were also a source of concern, as this precludes submission of some practical 
assignments through Turnitin, which thus increases the workload of the administrative staff who may 
have to email these files directly to markers. Finally, administrators noted that if it were possible to 
receive automatic notification of late submissions, this would reduce the use of administrator time 


taken in manually checking assignment inboxes. 
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Dependence on other people — Reliance on others during the assessment process was viewed as 
another potential point of failure. This referred mainly to academics not meeting the deadlines by 
which marks had to be returned to administrators. Academic engagement with online processes is 


often lacking and there were some challenges with students, who were not keen to use online tools. 


Manual, labour-intensive processes — Many references were made to an insistence on submitting 
hard copies of assignments, which was deemed unnecessary. There were numerous laborious, manual 
processes involved in the workflow such as inputting grades from spreadsheets received from academics 
into the student record system. Checking for plagiarism was also described as being labour intensive. 
Some administrators reported the requirement to scan feedback forms, manually enter marks and 
return hard copies to students. One administrator even reported that they were required to check 


word counts on assessments. 


Institutional procedures — Administrators noted a lack of clarity and consistency regarding standardised 
policy within the institutions; there was no common set of procedures across any individual institution. 
It appears that departments often manage the EMA workflows in different ways. This can cause alarming 
difficulties, as for example Turnitin reports are inconsistently viewed and interpreted. In addition, it 


was reported that the assessment responsibilities of job roles were not well defined. 


Legal issues — In addition to unnecessary cost and time-consuming activity, there was some concern 
about the data protection issues around providing hard copy samples of assessments to external examiners. 
It was also noted that high similarity scores could go unnoticed and not acted upon if academics only 


mark/moderate paper copies of assignments. 


Concluding remarks 


The process we have used to gather the data discussed here cannot be described as a representative 
reflection of practice across the institutions as we cannot be sure what additional insight might potentially 
have been provided by non-attendees. However, from this quite large and diverse sample, we are able 
to determine that there is a wide range of EMA practices across the institutions and even within them. 
We have also noted that there is a continuum of EMA adoption from those who make use of online 
submission only, through to those who handle submission, marking and feedback to students almost 
entirely online. It is important to note that ‘almost entirely’ is a key point here, as there are key elements 
of the marking process which EMA systems currently do not adequately support. Overall, this exercise 
has proved valuable in understanding the issues from the point of view of the administrative staff, whose 
voices are less often heard in discussions surrounding assessment. Analysis of the event’s evaluation, 
which was conducted by anonymous survey, revealed a desire for a new local community to share 


practice more regularly and formally. 


Our research has indicated that there is a lot of variance in the practice of assessment administration, 
not just across the individual institutions but between different organisational sub-units. This lack of 
consistency makes it difficult for the institutions to source technical solutions that would support and 
enhance all desired approaches. Anecdotally, we have noted that differences in practice are often thought 
to arise because of differences between disciplines, although some of these differences may stem from 
‘tradition’ rather than pedagogy. In any case, a standardised approach to EMA across a diverse institution 
is not pedagogically desirable. However, a common approach at department level assists in maintaining 


processes that are streamlined, efficient, accurate and non-repetitive or duplicating. 
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Lack of training 

Academics having little/no training 
on how to use Moodle/Turnitin 
Anonymous submission not currently 
available on Moodle Assignment 
Technology unreliable 

Academics not meeting deadlines for 
returning the mark sheets to admins 
(2); reminders have to be sent 


Students receive grade marks late; 


fuels disgruntled students 


Regarding our opening questions to the participants, the prompt which generated a (perhaps 


surprising) volume of feedback was the one asking for identification of particular “pain points’: 


Too many processes 

Manual data entry of marks on 
student record system (SRS) 

Lack of clarity across the institution 
(policy); not pusing the same systems 
across the College 

If academics are not checking online 
submissions because they refer to 
paper copies, it is very likely that high 
similarity scores for plagiarism may go 


unnoticed and so not be acted upon 


Academics resistant to marking online/ Hane copy Samipics for-extemal 


: examiners — costly, time-consumin 
don’t want to use technology (2) eens an 8 


i schaophobedtuaents and data protection issues 


Inconsistent viewing of originalit 
Need to remove paper copies, using ' Ee fone 


Moodle for return reports due to submissions being 


downloaded to shared drive 


Overall, the preference of the administrative staff who took part in this exercise was that assessment 
activities are conducted online. It is not the use of online technologies per se that is generally regarded 
as the most significant pain point involved in EMA, but rather the necessity for offline ‘sub-routines’ 
to handle aspects of processes that fall outside of the available systems. The wider list of pain points 


can be categorised into three key areas: people, technology and pedagogic requirements. 


In terms of ‘people’, it was felt that academics should be encouraged to mark online to help streamline the 
workflow. According to the participants, there is still a strong preference among the academic community 
for paper-based marking. As well as online marking in Turnitin, downloading of electronic copies for 
offline marking (e.g. using ‘track changes’ in Word) is quite common, but creates some administrative 


overhead in returning marked copies to students. 


A further strategy in common use is dual submission (that is, both online and hard copy), which can 
be a useful interim strategy when moving towards online marking. However, the participants felt that 
this halfway house of dual submission involves a double workload of managing two sets of submissions, 
and is therefore not suitable as a permanent process. Administrators’ preference appears to be strongly 
for marking to be done online; those who attended our event indicated that they are generally willing 
and able to support academic staff with this. Additionally, support improvements, such as more 
frequent or timely training sessions, were also recommended. We also observed a lack of consistency 


in relation to the academics’ and administrators’ areas of responsibility. 
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In terms of ‘technology’, this report provides evidence of a strong demand for automation between 
Moodle, Turnitin and student record systems. Some unusual practice has been highlighted here, which 
could be better managed by using Moodle and/or Turnitin for assignment processes. In addition, 
administrators felt that they expend excessive efforts in notifying academics of submissions, liaising 
between markers, reminding academics to mark and chasing for grades. It was thought therefore that 
systems could better support these processes to make them more automatic. Interestingly, occasional 
technical failures were not reported as a pain point. This may indicate an understanding and acceptance 


that technology is not always perfect, and a belief that it is still valued despite potential issues. 


In terms of ‘pedagogic requirements’, a number of issues were revealed. There was significant use of 
anonymous marking although it was not as common as we might have expected. Also, there were 
frequent requests for improvements to double marking (i.e. support for second marking, moderation 
and reaching agreement between markers) within Turnitin. We were surprised that some instances of 
checking for plagiarism (or rather suspicious levels of text-matching) were reported as being conducted 
by administrative staff. It appears that some administrators are very involved with aspects of the marking 


process such as assigning markers, checking word counts and even finalising grades. 


It was interesting to see the different approaches that the participants adopted in illustrating or explaining 
the EMA processes in which they are involved. If repeating the exercise described in this paper, we 
would want to ensure the participants are completely clear and explicit in their explanation of the 
assessment workflow including who does what. Also, we would ask for contact details on the activity 
worksheets so we could follow up with them after the event, as this would have enabled us to clarify 
some points or ask further questions. We deliberately chose not to do this at the time in case the 
participants felt uneasy about ‘naming and shaming’ their institution or department, but, in hindsight, 
we might have been able to provide better support if we had known who was experiencing the most 
difficulties. It would also be a useful exercise to survey academics regarding their experience of 
moving to marking online. A further study could look more closely at how administrators and 


academics interact with each other and among themselves throughout the EMA workflow. 
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Chapter 3: Assessment, Technology and Innovation 
in Distance Learning in the Bloomsbury Learning 
Environment Institutions 


Nancy Weitz!’ 
Kathy Seddon’! 


Introduction 


The brief for this review was to examine the use of technology in assessment in distance learning (DL), 
including Massive Open Online Courses (MOOCs), within the Bloomsbury Learning Environment 


(BLE) institutions and offer commentary and recommendations for future improvement. 


The majority of data was gathered in the form of hour-long interviews with lecturers, course directors, 
DL coordinators and learning technology staff within the BLE member institutions. Supplementary 


information came from questionnaires completed by them and others working in similar roles. 


While complete coverage of all DL courses and programmes was not possible within the scope of the 
review, a general understanding of current practices employed by consortium members enabled us to 
devise a model for the use of technology for assessment. This model provides the framework for 
discussion and, crucially, moves us away from a too-specific emphasis on individual technologies and 


towards a focus on good practice in learning design in relation to assessment. 


The analysis set out in the Assessment Toolkit, developed by Gordon, Hughes and McKenna (2015), 
offered a useful starting point for developing the model: “Assessment influences what students learn, 
the way they approach their learning and their attitudes towards learning and their course. Well-designed 


assessment tasks can also help prepare students for future employment or further study” (p. 5). 


The overarching context that we considered was that of postgraduate international courses, with a 
wide variety of students, a range of cohort size, and where the desired learner achievements have to be 
accomplished within a set framework of assessment. With this in mind we constructed a model for 
analysis of DL programme and course assessment of BLE member institutions which is informed by 


the suggested main purposes of assessment: 


e Assessment of learning, which occurs when teachers use evidence of student learning to make 
judgements on student achievement against goals and standards (summative). 

e Assessment as learning, which occurs when students reflect on and monitor their own 
scaffolded progress to inform their future learning goals (formative). 

e Assessment for learning, which occurs when teachers use inferences about student progress to 


inform their feedback and teaching. Peer interaction and assessment also occur here (formative). 


10 Nancy Weitz, Digital Learning Specialist, Architela. 
11 Kathy Seddon, Consultant. 
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The model we devised incorporates three approaches to assessment: 


1. A functional approach, in which compliance processes are met. Resources and results are 
generally available through technology. Summative assessment is more prevalent than 
formative assessment. 

2. An enhanced functional approach, where there is an increase in the use of formative 
assessment with tutor interaction and individual feedback enabled through technology. 

3. An innovative approach, with a strong collaborative pedagogical rationale and increased 


variety of learning activities. 


Approaches One and Two are hierarchical and additive in nature: One is the baseline; Two contains 
all of One with the addition of other, more formative features. However, Approach Three represents a 
different vision of assessment, in fact to whole course design — a vision that is more activity-based with 
collaborative features which serve to increase feedback to students. There is some overlap between the 
approaches, particularly One and Two, and some of these courses are beginning to bring in elements 


of Approach Three. 


Approach Three courses incorporate higher levels of formative assessment and they have a greater 
variety of assessment activities. Formative feedback comes not only from tutors, but also through 
peer-to-peer communication and collaboration. A high value is placed on formative assessment, peer 


interaction and feedback. 


According to the Assessment Reform Group (ARG), a task force set up by the British Educational 
Research Association, formative assessment or “assessment for learning’ is a key tool for raising 
achievement. Improving learning through assessment depends on the provision of effective feedback 
to students and on a recognition of the profound influence assessment has on motivation — students 


need to learn how to assess themselves and understand how to improve (ARG, 1999). 


We do not make any judgements about the quality of the assessment feedback in courses, nor do we 
claim that high quality feedback can only be accomplished through technology. However, we see 
technology as a key enabler for improvements in assessment practices in DL, and the way technology 


is used is often indicative of which approach prevails. 


The current situation 


To consider the three approaches, it is useful to match some of the courses included in our study to 
the approaches set out in the model to serve as examples. This is a subjective process, and while we 
feel that many courses have elements of more than one approach, most fall more strongly into a single 
category. Moreover, rather than listing in exhaustive detail every assessment element or tool, we 
present ‘snapshot’ examples that represent typical practice. This analysis is therefore intended as a 
starting point for discussion rather than a ‘set in stone’ classification or map of assessment taking 


place in BLE DL courses. 
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Functional approach 
The functional approach to assessment applies 


e when courses comply with university regulations in terms of assessment and course structure 
e when resources and results are generally available through technology 


e when summative assessment is more prevalent than formative assessment, which may not 


exist at all 


All of the DL courses within the scope of the review fulfil the minimum standard comprising Approach 
One to assessment: courses in this group comply with university regulations in terms of assessment and 
course structure. These may be legacy courses - originally correspondence courses where the material 
was formerly posted out rather than posted online. They are now mostly ‘paperless’ with assessment 


results available through technology. 


Summative assessment generally dominates, though formative assessment may also be present in some 
form. A high percentage mark is attached to the summative ‘unseen’ essay examination (not seen 
before the examination sitting): up to 80%. Multiple Choice Question (MCQ) tests sometimes form 


part of this grade. 


The rest of the assessment mark comes from coursework tasks which may include reviews, reports, 
essays, oral presentations, case studies, journals, logs, scientific and mathematical exercises, audiotapes 
or videotapes, depending on the subject matter of the course. Moodle and Turnitin are commonly 


used to access material and to upload assessments. 


At SOAS, the Centre for Financial and Management Studies (CeFiMS) has exams worth 80% of the grade, 
which are held in Examination Centres around the world. Coursework (20%) is generally online in Moodle. 
Huge numbers of students in over 160 countries participate. Centre for Development, Environment and 
Policy (CeDEP) courses allow two sessions of preparation prior to the exam, which again is worth 80% 
of the final grade; 20% comes from coursework. All submissions are now made online, for instance 


coursework PowerPoint presentations. 


Birkbeck runs four degrees for distance learning (with entry certificate routes). These are taught with full 
video streaming of lectures and are supported by digital lecture notes, email/phone/Skype tutorials. 
There are optional residential field classes and use of an electronic library. Many assessments do not 
require online interaction other than downloading the assessment. They do, however, make provision 
for electronic upload of standard practical assessments and also for essay style questions, requiring 
submission through Turnitin. This year they have started to use audio/video in formative assessment. 
The Department of Earth and Planetary Sciences offers all certificate and degree programmes via 
distance learning as well as face-to-face or blended routes; and while the final exam is mostly unseen 


essay questions, there are MCQ papers. Other Birkbeck courses are considered later. 


UCL IOE courses use Moodle and Turnitin to upload coursework assignments, such as self-recorded 
presentations, essays and quizzes. Feedback is available prior to final submission. GradeMark software 


is used for marking. 
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DL students taking two of the RVC’s MSc courses (MSc in Veterinary Epidemiology and Public Health, 
and Livestock Health and Production) sit exams held in local exam centres. There are also one to three 
coursework assignments per module uploaded in Turnitin with feedback by email. The Certificate 
in Advanced Veterinary Practice (CertAVP) has at least one module assessed by a short open book 
exam. Formative assessments include self-assessment quizzes. In the MSc in Veterinary Education, 


written assignments are submitted online and feedback given via Turnitin. 


LSHTM does not use Turnitin. The Infectious Diseases Programme (IDP) core has an unseen hand written 
exam which is scanned and marked online. Electives are submitted and marked online with comments 
from markers to agree grades. The IDP core assignment is marked and feedback submitted online. 


Global Health Policy final exams use exam centres and also require short essays with online feedback. 
Enhanced functional approach 
The enhanced functional approach to assessment applies 


e when there is an increase in the use of formative assessment 
e when there is tutor interaction with individual students 


e when feedback is enabled through technology 


Courses which exemplify Approach Two appear to be common in the DL landscape of the BLE institutions. 
Whereas Approach One is more or less based on the traditional form of the correspondence course, 
updated with modern communication tools for some of the delivery of (primarily summative) assessments, 
Approach Two courses show a greater recognition of the value of formative assessment and feedback. 


It is likely therefore that the majority of all DL courses in the BLE group fall into this category. 


Discussion forums in Moodle for peer-to-peer and peer-to-tutor interaction are standard in DL courses 
at LSHTM, as are Skype sessions with project supervisors and online seminar (Collaborate) sessions 
with groups. The MSc in Clinical Trials also offers online study materials with embedded questions 


and pop-up answers for automated formative self-assessment. 


All of the RVC course modules provide peer feedback forums, and some courses offer optional Skype 
tutorials. The CertAVP course offers self-assessment (formative) quizzes, in the form of either multiple 


choice or essay. 


Some innovations of the University of London International Academy (UoLIA) fall within Approach 
Two (some in Approach Three below), such as the custom built self-assessment plugin for essay type 
questions with model answers. Students have the option to self-mark their answers against the model 
answers. They have also introduced new support personnel, student relationship managers and a 


Student Advice Centre, which all provide feedback and monitor engagement. 


The lecture capture software Panopto” is used to record face-to-face lectures at Birkbeck, and Skype has 


been used for seminars for ten years. Formative assessment includes essays and mock exams with feedback. 


12 Panopto is commercial software used in many academic institutions to manage, live-stream, record and share videos. 
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SOAS CeDEP uses a Moodle tutor-led discussion forum. There is also unofficial student interaction. 


Skype is used for “student voice’ interaction with a designated tutor involved. 


UCL IOE uses a variety of forums, blogs and wikis, which are embedded in online activities; some are 
mandatory. There is some formative feedback in discussion boards and web conferences. Online tutorials, 
UCL assessments and reuse of module activities towards a final submission are used in the Education 
& Technology MA. 


Innovative approach 
The Innovative approach to assessment applies 


e when there is a strong, collaborative, pedagogical rationale 


e when there is increased variety of learning activities 


In identifying courses, or parts of courses, that might be deemed to exemplify Approach Three, the key 
diagnostic is the degree of student interaction and number of collaborative learning activities that occur. 
The thinking behind such activities draws on cultural theories (including social constructivism) and 
scaffolding for learning comes from tutors and peers in online activities. This underlying rationale 


influences both course structure and assessment. 


There were several courses in our study that went some way towards this kind of structure in their formative 
assessment. UCL Eastman Dental Institute’s Paediatric Dentistry MSc offers blended learning. The online 
interface (iPad Mini provided) allows staff to support students’ day-to-day work. Assessment is through 
written and practical presentations, essays and vivas. An innovative online logbook is provided as part 
of the programme; it records treatment and can be used to support further career progression. There 


is some degree of peer interaction. 


The new Birkbeck Geochemistry MSc utilises their online learning environment (OLE). Geochem students 
are provided with high quality photos of rocks (via Xerte tutorial) to analyse and comment on peers and 
tutors take part in the discussions that follow. Other assessment elements include unseen examinations, 
theoretical and practical assessed coursework, oral presentations and associated handouts, a research 


proposal and a dissertation. 


The Financial Sector Management course (MSc) has been developed by academics at SOAS’s CeFiMS 
working with UoLIA. Through their OLE, students communicate with an assigned academic tutor, 
administrators and other students on the course using discussion forums. The OLE is, however, mainly 
used to access study materials, to submit assessments, to communicate with tutors and support staff 


and to gain access to the University of London Library. 


Nevertheless, a complex module that is in development introduces a variety of activities, and activity- 
based assessments, into the course through a strategic simulation. Students carry out research which is 
then applied to a strategic case study. The case study will be supported by a simulation model enabling 
the student to build a business plan and carry out risk analysis and scenario planning. Students make a 
series of annual decisions to implement their business plan. At the end of each decision year, they will 


receive feedback on their progress and be able to reflect on their performance, comparing actual results 
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against their plans, and benchmark their performance against other students/teams. Students will 
work in teams of five to plan and manage their business, establishing long-term collaboration as a 


significant factor. 


The course which most fully embraces online interaction between students and tutors in its activities 
and assessments is the MA in Global Diplomacy (Online Learning) course developed by the Centre for 
International Studies and Diplomacy (CISD) at SOAS. The course demonstrates an holistic approach 
to assessment design where it is integral to course design and where assessment activities are mapped 
onto learning outcomes. Each module is assessed by five written online assessments (‘e-tivities’) 
comprising 30% of the module mark and one longer essay comprising 70%. The e-tivities provide 
formative and summative feedback to students as a means of monitoring their progress and 


encouraging areas in which they can improve. 


The concept of e-tivities, which underlies the course, was first set out by Gilly Salmon in 2002. An e-tivity is 
a framework for online, active and interactive learning that states clearly to the students its ‘purpose’, 
the ‘task’, the contribution (response type) and the ‘outcome’. She describes how learners should be 
supported through five stages in progressive participation in an online learning community: access and 
motivation, online socialisation, information exchange, knowledge construction and development. 


These are integral to this course design and assessment. 


Another promising development is UoLIA’s “Track C’, which is setting a foundation for more courses 
to move towards Approach Three. Track C is a technical framework or template for new courses and is 
designed to lead colleges towards change by getting Programme Directors to rethink how they approach DL. 
As some DL programmes are run by a small number of people with few resources to make big changes, 
UoLIA established the framework first and then looked for partners to develop new courses. This strategy 
arose in an effort to get control of content, establish coherency and increase efficiency to all programmes. 
UCL’s Professional Accounting course is currently running in the Track C framework, with group 


work (formative) in project teams running simulations. 


In addition, UoLIA has created a series of tools for mapping and tracking performance, which bring 
assessment into line with pedagogy. These include formative assessment questions (e.g. multiple choice, 
self-mark, etc.) which are mapped and weighted against learning outcomes. The Learning Outcomes 
Profiler (LOP) tool provides a single view to show progress against all learning outcomes within the 
module. Students are able to monitor and record their own perceived progress against key learning 


targets, and tutors can use the data to provide targeted support. 


The RVC is on the cusp of Approach Three with the MSc in Veterinary Education: for the teaching 
observation, students are asked to prepare a plan for a teaching session with all supporting materials, 
to video themselves delivering the teaching session and to write a critical analysis of the video. This is 
then submitted online and summative feedback is given via Skype. This activity shows an innovative 
approach to solving the problem of observation in a remote situation, but in order to fit within 


Approach Three, it could be revised to include formative, peer-assessed or collaborative elements. 


There could very well be other examples of Approach Three innovative practice happening within the 


BLE, which may come to light. 
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MOOCs 


MOOCs are special cases of DL courses. They are not included in the model outlined above primarily 
because they sit outside the assessment and accreditation requirements for qualifications and degrees 
that govern regular university DL programmes. As such, they are freed from the rigours of summative 
assessment. However, the novelty of MOOCs and rapid changes in what they are, what they can be 


and what they should be when administered by universities means they need to be watched. 


The very nature of their ‘massive’ and ‘open’ status makes the typical challenges of administering DL 
courses pale into insignificance by comparison. MOOC course design is tested by size, student range 
and global reach. Most MOOCs use basic multiple choice online tests to provide assessment, whether 
as formative ‘self-check’ mechanisms, or as the summative test at the end of the course to assign ‘pass’ 


or ‘fail’. Automation is key for managing the numbers and providing uniformity of result. 


When they do occur in MOOCs, activity-based formative assessment and interaction are generally 
peer-to-peer due to the sheer impracticality of tutor-student interaction. A large online discussion 


forum space is a standard feature. 


Forms of assessment in BLE institution MOOCs are somewhat constrained by what is available in the 
system being used. The University of London International Programmes (ULIP) offers MOOCs via 


the Coursera platform, which provides feedback through a range of assessment formats: 


e In-video questions (lightweight, ungraded questions presented within a video lecture) 


e Discussion prompts (discussion questions formatted as content items within a specific lesson; 
learners can respond to the prompt, read others’ comments and continue the conversation in 


the discussion forums) 


e Quizzes (supporting a range of question types including multiple choice, numerical response 


and text response) 


e Peer-graded assignments (open-ended assignments in which learners grade one another’s 


work according to a rubric provided by the instructor) 


e Programming assignments (machine-graded assignments that require learners to submit 


computer code) 


Lee and Rofe (2016) set out some original alternatives to current thinking on MOOC course structure 
and assessment. The course described in the paper is the product of a partnership between SOAS, 
providing academic expertise, and UoLIA, providing investment and project management, in liaison 


with Coursera as the platform operator. 


The course is structured around a series of e-tivities essential to learning research methods, each of which 
has a stated purpose, task, response and outcome. The course is based on a peer-led constructivist 

approach incorporating assessment for learning. Forums, central to self-reflection and peer support, 
were initially led by tutors, but, with 60,000 students, trained volunteer community mentors became 


essential. Small group discussions initiated by students used Google hangouts. 
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A community of learners developed where students tried things out with their peers before submission 
(the flipped assessment model). This model drew on elements of “paragogy’ (learning alongside peers) and 
the intellectual reflection (IR) approach that acknowledges and exploits peer learning. The course design 
moved away from ‘talking heads’ prevalent in MOOCs to recordings of researchers ‘in conversation’. 


The course thus provided learning opportunities that are not routinely captured by ‘completion’ metrics. 


In general, there is a wide variety of methods of assessment taking place in BLE DL courses. The context 
of the courses is extremely varied, particularly in terms of student numbers and governing regulations. 
The assessment model may become a helpful tool to understand other DL courses in the BLE and 
determine ways to raise Approach One courses into the Approach Two level, or whether to plan for 


more extensive redesign in the aim to reach Approach Three. 


Issues and Opportunities 


The overall impression we gathered in our investigation was one of desire for change and improvement 
but a recognition of the difficulties and frustrations of actually implementing change. The complex 
web of relationships between the BLE institutions and the University of London can lead to both 


opportunities for and obstacles to change. 


This section includes the opinions and experiences of staff working within the institutions, and our 
observations on those perspectives, leading to recommendations. In the interest of anonymity to 


protect respondents, names and institutional affiliations have been omitted. 


Not surprisingly, there are a number of common concerns across all the BLE, and these have been 


brought together under specific category headings: 


e Communications and sharing: across and within institutions 

e Skills, support and training: technical and online learning 

e Assessment and activity design: aligning assessments to good learning design 

e Cost and resource: including time and manpower 

e Authenticity and identity: addressing verification and plagiarism 

e Technology and access: including complexity of systems and internet access by students 
around the world 

e Course size and student profile: how these affect course design and assessment 


e Attitudes towards DL and online learning within colleges 
Issues 


Generalised here as ‘issues’, these experiences and views reflect the frustrations, problems and 


concerns around assessment of staff currently working in DL in the BLE institutions. 


Institutional communication and sharing — ‘Siloing’ is a key problem for many of the institutions, 
in that there is a relative lack of sharing of ideas, resources and practice especially between institutions, 
but even within them. One institution described how there is little knowledge sharing across departments, 
and there appears to be no central role for overseeing, or even just being aware of, DL courses or practices 
within those courses. Even where intra-institutional communications and oversight are present, the 


sharing of practice between institutions is limited. 
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Skills, support and training — Most of the institutions in the review remarked about skills gaps in both 
students and teaching staff and the difficulty of training and supporting remote students, tutors and 
off-site markers. Of concern are both general technical abilities and those skills specific to using course 


technologies and working/learning in an online environment. 


Assessment and activity design — Assessment design is (or should be!) intrinsically linked to course 
design. One of the problems and frustrations we encountered was a recognition amongst staff that many 
DL courses suffer from their legacy as ‘correspondence’ courses and need to be updated to benefit 

from more learner-centred design and take advantage of online tools - to move from their position in 


Approach One into Approach Two, if not to be redesigned completely as Approach Three courses. 


Others expressed a dissatisfaction with the way e-learning is experienced online and the difficulty of 


navigating around the course website and locating activities. 


Taking into consideration access to technology (see “Technology and access’ below), there can be a 
conflict between designing rich, activity-based online learning and making learning accessible to 


international students in the field or in areas with poor internet reliability. 


There is also a frustration that despite efforts to develop engaging activities, a number of students do 


not participate. 


Development cost and resource — Following on from the previous point, creating engaging assessments 
and activities is labour-intensive and calls for certain expertise that may not already be in the skill set 
of the staff. There may also be additional costs associated with development of such materials. The 


cost areas affected are: 


e budget 
e time 


e@ manpower 


These are the three areas that are generally already in short supply in higher education. 


That said, the existing assessment regimes, particularly for managing summative assessment, are very 


labour-intensive, and respondents expressed a need for new systems that streamline processes. 


Candidate authenticity and identity - Common to all the institutions is the concern for authenticity 
in the assessment practice in DL, particularly when moving to online assessment. The ability to verify 
identity and have confidence that the work submitted is indeed produced by the student being assessed 
are fundamental issues. It is of high importance that the colleges and the University of London retain 
their reputation as degree-granting institutions with high standards. Plagiarism, cheating, false identity 


all potentially undermine the value of that degree. 


Course size and student profile — A wide variation in the number and demography of students in DL 
courses is a common challenge. Some issues are addressed in other headings in this section, such as 
available technology and internet access in countries across the world, but there are many other factors 
that make it difficult to serve a large, dispersed student population, including culture, language, age, 
life experience, career level, time zone and sheer numbers. In many cases, this situation extends to 


tutors and markers, who may also be widely dispersed. 
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Technology and access — Internet and data security are a concern amongst technology specialists. 


Some expressed the worry that too many people have access to sensitive data. 


Others who administer the courses find the assessment technology has ‘glitches’ or is not completely 
fit for purpose, in some cases having been developed without understanding the learner experience or 


the needs of those giving feedback or marking. 


Another issue, related to student profile (see “Course size and student profile’ above), is the availability and 
type of technology and internet access for students in other parts of the world. The prevalence of mobile 
devices and relative lack of desktop computers makes online learning that is not optimised for small screens 
difficult - sometimes impossible — to use. Similarly, unreliable and slow connection speeds can hamper 


the ability of students in remote areas or who need to work in the field from accessing their courses. 


A proliferation of different online systems across the University of London and colleges, many of which 
are not interoperable, creates a drain on resource, time and personnel. In some cases, it can also create a 
dangerous reliance on the individual(s) who built the systems. In one college, there is only one person 
who can maintain a complex, homemade assessment management system. If he or she is absent for 


any reason, there is no safety net. 


Attitudes — Some staff members witness a lingering attitude in departments or colleges that DL is 
considered a lower priority or less worthy than traditional on-campus learning, which leads to under- 


resourcing as well as low morale. 


Other attitude problems are apparent with colleagues who remain sceptical about the value of online 
learning more generally. This can also drain morale and continually undermine attempts to expand 


online offerings and to innovate. 
Further opportunities and recommendations 


‘Opportunities’ here refers to the views of those working in DL, as well as the cross-college commonalities 
and possible ways forward that we identified. Some of this takes the form of observation together with 


some suggestions and recommendations for change. 


Institutional communication and sharing — The BLE is a solid start to sharing practice and resources. 
Similar initiatives could be established with the aim of building up assessment design skills across 
programmes and institutions. The expertise displayed by staff running innovative courses, as seen in 
Approach Three, could become the basis for hands-on practical workshops on formative assessment 


design, which could take as their guide The Assessment Toolkit (see below). 


Other DL courses may be using innovative practice in BLE institutions, but these were not apparent in 
our investigations. A job role that includes developing a clear overview of DL across the consortium 


would provide the foundation for dissemination and sharing. 


Skills, support and training — Support and training in courseware and online learning for both staff 
and participants (students) are frequently afterthoughts to any kind of online learning in any sector. 
They are the factors most often under-planned and under-resourced, but which causes so many problems 
for the success of programmes. One benefit to centralising key systems is the ability to centralise 


training on that system. 
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One of the more significant developments in relation to supporting students in online learning is coming 
from UoLIA, who are adopting an approach to surround courses with dedicated student support teams. 
This has the benefit of providing a reliable first point of contact for any problem a student is facing, 
performing triage and then finding the right mechanism to get the specific support needed. This avoids 
the common problem of students barraging their tutors with technical problems and questions about 


extraneous issues such as exam policies, etc., leaving the tutors to concentrate on teaching. 


One idea, to address the need for more general skills associated with learning online, is to expand on 
RVC’s pre-study, orientation-style mini-MOOCs for prospective students to give them time to become 
acquainted with the specific technologies used in the full course. These can also address library use, 
academic writing and a wide range of other skills that students will need before they start the course. 
Postgraduates require many of these same skills and, particularly, need to familiarise themselves with 


the culture and workings of the institution they are entering. 


Similar attention should be paid to training and supporting staff in all areas of online learning: 
courseware technology, learning technologies and media (i.e. video), online tutoring and facilitation, 


and assessment design. 


One thing to keep in mind is that training and support are not the same thing. Training is proactive: it 
involves defining lessons and actions that build towards a particular goal (i.e. learning how to use the 
online system). Support is reactive: it steps in to address questions, problems or needs expressed by 
the user. Training is a visible provision, while support, with its ad hoc nature, is not so visible. Unless 


participants complain about not getting support, the deficiency can go unnoticed. 


Assessment and activity design — Certainly an overhaul of existing courses to update the assessment 
and activity design is a labour-intensive and time-consuming job. However, with the relaxing of strict 
exam regulations announced by the University of London earlier this year, we recommend that colleges 
undertake a review of assessment in their DL courses with a view towards increasing formative 
assessment and student interaction. This is already happening in some of the colleges and should be 


extended across the board. 


Many new opportunities are opened up by the change in assessment policy, but caution is also required in 
order to avoid overwhelming students and tutors with a raft of exciting, new but unnecessary assessment 
activities that mainly serve to increase workload. Rather than adding more formative assessments into an 
already rigorous course, it is better to replace work done one way with work done another. Scaffolded 
assignments are a good way to shift existing appropriate workload towards sensible deadlines and 


timely support and intervention. 


Where there is already high quality feedback using traditional methods, a shift to electronic delivery 
should not erode quality, but this would need to be monitored and supported while staff become 


accustomed to change. 


UoLIA and some colleges are planning for or already bringing in progress tracking, and there is 


discussion underway about introducing peer assessment. 
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Generic activity templates for various types of formative assessment, complete with mapping to 
learning objectives, could be developed for use and reuse across the BLE. These would save time and 
provide guidance for course developers and tutors. As a starting point for this, the Assessment Toolkit 


detailed below has a table illustrating a mapping of different assignments against learning objects. 


The Assessment Toolkit — A significant piece of work that should have wide application across the 
University of London was published in 2015. The Assessment Toolkit, co-authored by Claire Gordon, 
Jane Hughes and Colleen McKenna for the University, was designed to support the review and redesign 
of existing courses as well as the development of new ones for ULIP. The toolkit does not preference 


particular assessment methods and offers a thought-provoking tool for reflection. 


The toolkit is built on the principles that assessment should be valid, reliable, fair and inclusive. The 
first table provides a comprehensive breakdown of how assessment tasks might fit learning outcomes 


in courses. Sections on assessment design that reduces misconduct are helpful. 


Use of a range of assessment methods is then detailed. This includes: bibliographies, blogs, computer- 
based assessments, essays, MCQs, oral presentations, policy papers, portfolios, posters, short answer 


questions and other online assessment activities. 


When looking into combining assessment methods, many factors such as accessibility, scalability and 
range of feedback formats are considered and a useful mapping exercise is set out. The Masters in 


Global Diplomacy, SOAS, is used as a case study for this. 


Marking and feedback are reviewed in detail and questions are offered to allow reflection on the approaches 


that course designers might use. The appendices offer a comprehensive list of other similar resources. 


Our recommendation is that this resource should be considered essential reading, and efforts should 


be made to support staff in using the guide to improve the formative assessment in their DL courses. 


Development cost and resource — Shared practice can reduce costs and resources. In addition to 
utilising the templates in the Assessment Toolkit, the BLE could provide a central bank or library of 
activity designs, technology, video clips and other plans, tools and materials, which can prevent the 
ubiquitous problem of ‘reinventing the wheel’ for every new course or module. Commonalities of the 
virtual learning environment (VLE) platform across institutions can reduce the workload on support 
staff and shared resources. Content, module plans and instructional design tools (i.e. learning 
objective maps) that build on the Toolkit exemplars can save course director and tutor time through 


duplication in development. 


UoLIA’s Track C framework and assessment mapping tools also appear to offer a productive way 
forward for new courses, whereas existing courses could benefit from ad hoc sharing and combining 


efforts as discussed above. 


Implementing sensible systems for online delivery and management of summative assessment could 
ease the workload for staff who are currently burdened by this problem. That, in turn, could free up 


time to work at increasing formative assessment. 
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Candidate authenticity and identity — Ways to combat fraud and cheating are being researched within 
the institutions, and if there isn’t one already, it would be worth putting together a task force to explore 


the latest tools and services for assessment, building on the suggestions in the Toolkit. 


This problem has, in the past, been a legitimate obstacle for wholesale adoption of online assessment, 
but that is no longer the case: a number of companies have stepped into the void to provide validation 
services and administer safe exams online (such as ProctorExam, Software Secure and Assessment Systems 
for Good Measure, etc.). Whether or not the University of London and/or BLE institutions wish to 


buy in a service, they need to understand new opportunities to administer secure assessments. 


Course size and student profile — According to respondents, there is some indication that DL student 
enrolments are up in BLE member institutions. With this positive development, it will be important to 
prepare for scaling courses for greater numbers and accommodating the variation in student populations. 
Lessons may be learned from the experience of running MOOCs (see below), and some solutions to 
problems with technology may be addressed by using more ubiquitous online systems. Moreover, 


assessments can be designed and modified to fit the needs of the students. 


Student needs can be monitored through requesting frequent feedback that targets particular issues and by 


implementing technical data collection and analysis tools within the courseware (i.e. page views, statistics). 


Technology and access — Technology design should be user-focused, with design and testing performed 
in collaboration with representatives of all user groups: students, tutors, markers, admin staff, etc.. 
Some institutions are using online forums to provide a platform for the student voice, which is a good 
step forward. The tension between standardisation and individual needs is ever-present and seems to 
make compromise a constant. However, there may be times when the apparent drivers for needs are 
more about tradition or arcane policy than about true necessity. In (re-)designing systems, it is worth 


getting to the root of anything that surfaces as a conflicting need to see whether 


e itis really driven by absolute necessity 
e itcan’t be accomplished by a completely new approach 


e another college or department has solved a similar problem 


If there is effective technology available that is underused (i.e. GradeMark, audio feedback, web conferencing, 


etc.), efforts could be made to encourage use through workshops and one-to-one advice sessions. 


In the case of multiple systems for assessment and marking, simplification and standardisation, wherever 
possible, would increase efficiency, save costs and decrease demands on staff. Training and support 
materials and regimes can then be centralised and shared across the University. Greater capacity for 
support and maintenance is ensured, by increasing the number of staff (or external support communities) 
who understand the system, and this in turn enables colleges to avoid being left paralysed by the loss 


of a single staff member. 


We recommend that development or buying-in of individualised technology with limited application 


be phased out except in very specialised circumstances. 
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Attitudes — There is no absolute solution to the problem of poor attitudes to online or distance learning. 
In either case, the fight tends to be against a retrenched conservatism rather than a specific foe that can 
be attacked head on. There are only three approaches that seem to have any effect, and — ideally — they 


will be employed simultaneously: 


e tireless ‘hearts and minds’ campaigns that continually show off good practice, good results 


and high standards; 
e buy-in from people in high places who can lend their power, authority and standing to the work; 


e attrition: the world is moving more towards remote and online learning. Those who object 


will increasingly be out-voted, converted or retired. 


MOOCs — MOOCs can be the agents of change (whether embraced with eagerness or dragged to with 
reluctance) for more traditional DL programmes. The SOAS/UOLIA course outlined by Lee and Rofe (2016) 
shows that MOOCs have the capacity to provide a testing ground for innovation. If interactivity and 


peer feedback can work with massive student numbers, they can work in a typical DL course. 


One word of caution, however: when a MOO is used as a taster or marketing tool to increase enrolment 
in a full course, the student experience in that MOOC will affect the expectations of students who go on 
to take the full DL course. Any innovations introduced to make the MOOC engaging, such as variety 
in formative assessment, will be accepted by the students as a true reflection of the full course to come. 


To retain student satisfaction, the full course will need to carry through the experience of the MOOC. 


Conclusion 


In this report, we have frequently moved beyond a strict adherence to the topic of assessment to 
include wider issues of DL and even online learning more generally. This is because assessment, when 
approached effectively, is intricately entwined with other aspects of learning: assessment is itself an 
aspect of learning. Moreover, the issues and opportunities we identified and explored through the 
filter of assessment and technology are common to the wider picture of online learning. Assessment 


for learning is part and parcel of design for learning. 


While we don't claim to have silver bullets to address the issues or solve the problems identified by 
BLE DL staff, we hope that this report and the opportunities we suggest may provide an objective 
viewpoint that shows up common ground between institutions, affirms individual perceptions and 


experiences and provides some new ways of looking at assessment in distance learning. 
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SECTION 2 


Case Studies — 


Alternative Tasks and Formats 


Chapter 4: Blogging and Social Media for Formative 
Assessment in Marketing and PR Modules 


Deborah Grange” 
Wendy Hein” 


Summary 


These modules develop formative assessment practices that facilitate critical and transformative learning. 
In particular, this approach focuses on students using their own experiences as case studies, which are 


then analysed via academic theories and frameworks. 


Implementation 


This case study considers assessment practices across two modules taught in Birkbeck’s Department of 


Management. 
Blogging in a marketing module 


Students are asked to keep a blog on a topic related to their own consumer behaviour for formative 
assessment. Past examples include Korean pop music and the iPhone. In a series of blog posts, they 
associate the various theories to their practice. The academic then discusses the blog entries during 
seminars as live case studies. Students themselves also choose specific blog posts and then analyse and 


evaluate them with reference to the research literature. 


In a separate formative assessment, students select a behaviour that they would like to change over the 
course of a month. For example, some students choose to stop smoking; others decide to follow a fitness 
programme, etc. They blog about their progress (or lack of) on this endeavour, referencing the academic 
literature, including theories of behavioural change. This personal experience of systematically effecting 


change equips students with a direct and nuanced appreciation of the challenges of marketing. 


The blogging software used was either Blogger or Wordpress, both of which are free. 
Social media in a public relations module 


Part of the assessment requires each student to choose a company as a case study to follow over a nine- 
week period. Students are encouraged to set up Google Alerts to keep abreast with any news on the 
company and to engage with both online and offline resources. Again, this is an example of the use of 
technology to provide students with original research and live case studies. Students are advised to 
choose the case study carefully because some students have later gained employment with the company 


they have chosen for this assessment. 


13 Case study author: Deborah Grange, Senior Learning Development Tutor (TEL), Centre for Transformative Practice in 
Learning and Teaching, Birkbeck, University of London. 


14 Project lead: Wendy Hein, Lecturer, Department of Management, Birkbeck, University of London. 
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Benefits and Challenges 


Not all students kept blogs, as they were not formally assessed. It worked very well for those students 


who did choose to keep a blog. Indeed, a few students have now entered careers as bloggers. 


Take-Aways 


Students are advised that using blogs for formative assessment tends to work better for class numbers 
of 70 or less. A later cohort had over 100 students which means that blogs are kept by groups rather 


than individuals and it is possible that interest may decline. 
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Chapter 5: Blogging for Summative Assessment in 
Postgraduate Education 


Tim Neumann’? 


Summary 


Blogging is widely regarded as a useful tool for reflection, articulation of ideas and learning from peers 
— activities that support the formation of knowledge. Hence, blogging is often used in the formative stage 
of a module. Entries normally address curriculum topics for a particular week, so peer or tutor feedback 


works best with quick response times, before the curriculum journey moves on to new topics. 


Such a body of writing of course only develops with regular and consistent engagement. While engagement 
can be enforced by making a blogging activity mandatory, students would be better motivated if they 
received a more tangible return for their efforts. High quality formative feedback would provide such 
return, but at the cost of a heavier tutor workload. Embedding the blogging activity into the summative 


module assessment would provide an alternative motivation without overloading tutors. 


Over the course of a module, students can accumulate a significant body of writing in their blogs, though 
this writing may be too raw and unfocused for summative purposes. Nevertheless, it can act as valuable 
evidence for a student’s learning journey, which can be moulded into a piece of writing more suitable 


for summative purposes. 


This case study reports on how a blogging activity can be used to accumulate evidence for the purposes 
of summative assessment by getting students to blog regularly and use their writing to critically self- 


assess their learning progress. 


Implementation 


The activity runs in a fully online MA module on Education and Technology with annual cohorts of 
between 30 and 60 students. Students are asked to post regular blog entries in response to weekly prompts 
that are linked to each week’s curriculum topic. Students have the choice to post their entries either in 
private journals that are only visible to them and the module tutors, or they can post to class blogs, 
which are visible to all module participants, who can then comment. While the private journal loses 
the peer feedback element, it provides students with an option to record their thoughts privately, 


which is sometimes appropriate for a variety of reasons. 


Technically, Campus Pack, which is an embedded component of the UCL Moodle installation, was 
used for both the private and the class-wide blog. Compared to other blog systems, Campus Pack 
features fewer design options and functions, but as the focus of the blog entries was the actual writing, 


Campus Pack was found to be very suitable. 


Module tutors do not formally monitor any blog entries, but can do so if they wish and find time. For 


the summative submission, students do not directly submit their complete set of contributions, but a 


15 Case study author and project lead: Tim Neumann, Learning Technologies Lead, UCL Knowledge Lab, UCL Institute of Education. 
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1,000-word reflection of their learning journey within the module, with the requirement to use excerpts 
from their blog entries and, if relevant, discussion forum posts. This means that students must construct 


an academic argument using their own contributions as evidence. 


The 1,000-word reflection counts 20% towards the overall module grade. Students submit a draft at 
the end of the module’s teaching phase and get formative comments on their writing before the final 


summative submission. The blog itself is not marked directly. 


Evaluation 


This activity has run for over five years, and is not evaluated separately beyond the standard evaluation 
mechanisms, such as end-of-module questionnaires and participation monitoring via the blog’s 


usage statistics. 


The reflection submissions themselves often contain comments about the usefulness of this activity, as 
it helps students realise and articulate how their understanding has developed over the module period 


of ten weeks. 


Statistics show that the private blog option is roughly twice as popular as the more open class blog. 
Unsurprisingly, the class blog produces more comments, although with a post-to-comment ratio of 3:1, 


the class blog does not create a high level of genuine discussion. 


While weekly blog posts were originally declared as mandatory and required to pass the module, the 
statistics have shown that only seven in ten students produced or exceeded the required minimum 
number of blog posts. The mandatory status was therefore adjusted mid-module. However, as it is in 
the students’ interest to write blog entries in order to be able to complete the summative reflection 


piece, every student has contributed something to the blog. 


The average number of views per public blog entry was just above 23, which suggests that the entries 


were read by a number of students, so at least a passive idea exchange appears to take place. 


Benefits and Challenges 


Tutor feedback on every individual blog post would be difficult to evaluate. Therefore, in this activity, 
the tutor only comments on the final drafts of a reflection piece constructed with evidence from blog 

entries, followed by summative marking of a final submission. Consequently, this activity can run with a 
higher number of students, while retaining the benefits of blog posting in terms of regular articulation 


of developing knowledge and the exchange of ideas. 


By using blog posts as evidence, students learn how to work with text as data, and they develop their 
academic writing skills by creating arguments. Students tend to choose a theme around which they 
organise their reflections, and the final submissions are as much documents of a knowledge development 
process as investigations of a curriculum topic, in particular as students are also encouraged to 


supplement their own observations with existing literature. 
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The activity is not without challenges though. Even though weekly blog entries are mandatory and excerpts 
from the posts are supposed to be part of the summative assessment, students need prompting beyond 
the pre-prepared weekly instructions. Many students still fall behind or simply do not complete the 
required number of posts, which tends to have direct consequences on their marks, as they only have a 
limited amount of data to work with. The conscientious students who post regularly on the other hand 
are aware that their weekly posts are already contributions to the overall module assessment, and they 


tend to achieve much better results. 


The separation between a private and a class-wide blog is artificial. It would be desirable to just have a 
single blog that allowed students to set the viewing permissions for each post individually. Blogging 
platforms such as Wordpress have this functionality; however, a dedicated advantage of Campus Pack 


is its deep integration in the virtual learning environment, so students do not need to switch platforms. 


Take-Aways 


It is important to use the right tool for this activity. A limited functionality is beneficial in this case, as 


this reduces potential distractions; however, a setting to control visibility for each post would be 


highly desirable. 


Access to the blog needs to be as easy as possible. It is already a challenge to encourage students to 
contribute on a weekly basis, and any technical barrier, such as switching platforms or even having to 


log on elsewhere, would exacerbate this issue. 


A mandatory requirement for the weekly blog post is a must, although tutors should be prepared for 
concessions to prevent outright failure based on not keeping up. While this activity has some embedded 
incentives, including preparing elements for the summative assessment during the taught phase of the 
module, these are not enough to ensure regular contributions. Tutors must remind students of the 


importance of regular contributions throughout. 


Splitting up final module assessment into multiple components can be a challenge, but as students 
produce a coherent piece of academic writing, the standard grading criteria that are used for similar 
pieces of writing (e.g. essays) can be applied. If assessment is based on contributions other than longer 


pieces of writing, grading criteria need to be considered. 
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Chapter 6: Using Journals in the SOAS International 
Human Rights Clinic 


Lynn Welchman’° 


Summary 


Clinical legal education (CLE) is based on ‘learning-by-doing’ and praxis: reflection on what students are 
doing, and what and how they are learning is a critical part of CLE methodology. After two years of 
structuring reflection into class time and in the preparation of students’ Project Portfolios, I introduced 
the ‘Project Diary as an annex to the Portfolio. The aim of this was to provide an additional, regular 
route for written reflection during the course of the project work; and to supplement regular oral 
Project Rounds in the Clinic plenary and discussions in our team meetings. Originally, it was prepared 
and submitted by the students as a Word document, but in the last four Clinics we have used the 
Campus Pack Journal facility in Moodle (supplied by Learning Objects). The Diary will be formally 
assessed from 2017/18. 


Implementation 


Project Diaries are set up by the SOAS e-learning department, who create a Project Diary for each student 
in the project teams to which they have been assigned. The team work on a research and advocacy project 
with a human rights non-governmental organisation; I usually have five or six project teams working 
with different external partners in each Clinic. The Diaries are set up as soon as I have assigned the 
students to their teams (in early October), and they remain live until the Project Portfolios are submitted 
(in mid-February). I explain the learning purpose of the Diary and the place of reflection in CLE praxis, 
and encourage students to start making entries immediately and, as far as possible, regularly. Students 


cannot see each other’s Diary posts. 


The Project Diary of each student is annexed to the Project Portfolio submitted by each student for 
assessment (at 40% of the course mark). The Diary is read by markers (along with the completed project 


document and a self-evaluation statement from the end of the first term) as an annex to the Portfolio. 


Evaluation 


Inevitably, some students make more frequent use of the Diary than others, and from 2017/18 I will be 
formally assessing the Project Diary at 10% of the student’s course mark, within the new postgraduate 
Credit Framework that SOAS is implementing from 2017/18. 


16 Case study author and project lead: Lynn Welchman, Convenor, International Human Rights Clinic, School of Law, 
SOAS, University of London. 
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Benefits and Challenges 


Benefits 


As supervisor of the different projects, as well as convenor of the Clinic, I am able to keep track 
of progress on the project in each team and with each individual team member in between my 


regular team meetings and the Project Rounds in class. 


I ask the students to log the approximate time spent on different tasks associated with the 
project, including team meetings and exchanges with the project partner, challenges faced, 
legal developments, their research activities and challenges. This is a good practice and skills- 
building for the students, as well as a useful way for me to feed in ideas, sources etc. on the 


project work. 


Iam able to pick up on any tensions developing in the team or the project more generally; 
while the latter might be voiced in Project Rounds, the former are not, and the Diary provides 
a route for the individual student to flag issues to me that they might not be comfortable 


voicing in a meeting, but that might be affecting their learning experience. 


Some of the Diaries become quite engaged as the intensity of teamwork and confidence in the 
Clinic setting builds; students get used to the idea of writing Diary entries and reflection on 


the learning process gains its own momentum. 


Challenges 


I try to read each Diary every fortnight; it should probably be once a week, but student numbers (24 in 


the last Clinic) make this a challenge. Engaging in a conversation with each student (by responding to 


their Diary entries) about how they are reflecting on the progress of their project and their learning 


experience, and the challenges they are encountering, is quite an intensive process. 


Take-Aways 


My experience suggests it is useful to have the journal formally assessed, but as the next academic year 
(2017-18) will be the first time I do this, I do not yet know whether and how it will change content. 


This form of assessment is quite demanding for the academic supervisor. 
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Chapter 7: Running a Group Assessment in Mahara 


Mira Vogel’” 


Summary 


Mahara" is an e-portfolio platform with an educational focus. With support, Mahara works as a flexible, 
multimodal platform for group assessment. As a multimedia-enabled platform for student work, it is 
particularly valuable where there is need for a level playing field. It is important to be clear up front 
that students are conscious of the education-specific nature of Mahara and may well raise Wordpress,'” 
with its huge global usage, as a better platform in employability terms. However, if you do not have an 


institutional Wordpress installation, Mahara serves well. 


Groups of students were required to produce digital exhibitions for assessment. The exhibitions needed 
to include a range of different audio-visual and social media, and students needed to be able to control 
how these were organised and encountered by their viewers. Staff needed to get students into groups 


with minimal fuss and confusion, apply a deadline fairly and give access to assessors. 


implementation 


The pioneering department’s teaching committee made a decision about platforms. After looking at a 
comparison of different possibilities (https://goo.gl/mO0etXl), they chose Mahara because it was flexible 


and would be supported by a responsive team with an educational remit. 
There are a few things to flag up about using Mahara for group work: 


Support students to work as a group — Unlike a wiki, Mahara can only give very patchy information 
about who did what in a group. If you want students to cooperate and influence each other rather than 
simply dividing the work into individual territories, you will need to explicitly guide them to do this. 
Mahara’s Page Comments are one way that students can influence without directly editing each other’s 
work. The Group Forum (which needs to be set up by an Administrator) is a possible site for a group 
to externalise decision-making, although third-party platforms dedicated to consensus building, such 


as Loomio,”? may work better. 


Choose the best time to induct students — Do this hands on and in groups, and not too early so 
they have an idea of what they would like to do and are ready to make and enact some decisions 
together. We provided an online worksheet which was easy to navigate out of order, and encouraged 
the students to start on their projects rather than slavishly working through the exercises. A couple of 


experienced Mahara users were on hand to answer questions. 


17 Case study author and project lead: Mira Vogel, Digital Education Advisor, UCL. 
18 https://mahara.org/ 

19 https://wordpress.org/ 

20 https://www.loomio.org/ 
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Get students into groups on Mahara in advance — This saves precious time and avoids disorientation. 
Where there are many groups, a Mahara administrator can do this with three separate CSV uploads: 
the first establishes students and colleagues as users (if they have never logged in previously); the second 
establishes the groups; the third populates those groups with those users in their respective roles. See 
https://wiki.ucl.ac.uk/x/3YCKAw for guidance. 


Give all students the same Group role - Member is probably best — Originally the Group ‘About’ 
page seemed a logical choice of home page since this is the landing page when you click a link to the Group. 
However, in order to edit that “About’ page, Group Members need to have a Group Administrator role 
— but this in turn creates hassle since Administrators retain editing rights after the deadline. It is far 


better for students to have a Member role, and giving them all the same role is more equitable. 


Keep two members of staff as Administrators in the Group — Groups need at least one Administrator, 
and two is safest. Be aware that the presence of staff may inhibit students from drafting and 


experimenting there, unless steps are taken to reassure them and set out expectations. 


Let students know how they can contribute work — Students can contribute to a group in two contexts. 
They can either work within the Group context, in which case the work becomes editable by all other 
Members; or they can Share work created within their own Portfolio. Assuming they have made this 
copiable, this latter way allows them to retain their original while also allowing the Group to work on 


the copy. However, the former way is more usual, because it is more obvious. 


Applying a deadline — Assuming students have a Member role, then in the Group’s Settings a Group 
Administrator can set the Editability to the required deadline. 


Making an assessor-friendly space — Group Members gather all Pages into one Collection. The Collection 
allows a Navigation block (menu of pages) to be generated, and this can be placed on each Page in the 
Collection. Collections also allow Members to share all the Pages at once, which is another way to give 
access to others. The first Page of any Collection is the landing page for that Collection, and students 


should be supported to make that effective as an orientating, inviting first impression. 


Make sure that assessors can easily find the work — The failsafe way is for a Group Administrator to 
add the assessors to the groups whose work they are assessing. Mahara does not let groups submit work 
for assessment — only individuals can do that - so an alternative approach is needed. If the assessors are 
not added to the groups, then permissions management is required (because students and staff are 
usually reluctant to make pages public before assessment). This requires students to follow a further 
set of instructions and staff to check to make sure they have followed these correctly. So you can see 


why, if assessors are part of the Group, things become much easier. 


We have guidance on all the above aspects in our Guides for Particular Modules section of our 
MyPortfolio Resource Centre released under a Creative Commons Attribution Share-Alike licence: 
https://wiki.ucl.ac.uk/x/DwXxAQ. 
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Evaluation 


We have evaluated this use of MyPortfolio in two contexts using questionnaires and focus groups, 


observation during induction, and opportunistically asking students. 


Benefits and Challenges 


Mahara is a highly flexible environment - staff are often delighted with what students produce there 


and award high marks. Students with less technical experience appreciate Mahara. 


However, students were split on whether or not Mahara allowed them to present their materials as they 
wished, and tended to disagree over whether Mahara allowed them to express themselves more 
meaningfully than in an essay (unsurprising since these students had gained admittance to their 
course of study on the strength of their essays). To address this we would recommend making your 
best case to students for multimodal work. On the technical front we would recommend stripping 
down the theme to as close to a blank canvas as possible, and ensuring that students are inducted to 


the full range of possibilities in Mahara and have a range of examples to refer to. 


Mahara is not widely used outside education, and you need to be prepared to justify its use to students. It 
is also felt to be quite constraining for technically advanced students, so be clear about the assessment 
criteria and emphasise the importance of a level playing field for a fair assessment. If you want to reward 
technical acumen, allow students to depart from Mahara. You may need to repeat these justifications 


for the duration of the projects. 


Out of the box, Mahara is not primarily a collaborative environment, although it is an effective 
cooperative one. In other words, students can contribute their bit, discuss in a forum and comment on 
each element, but Mahara does not store versions for comparison, nor does it record contributions in 
a fine-grained way. So if you want to find out more about who contributed and influenced the work, 


you will need to rely on students’ own accounts of what they did. 


Students were more likely to resolve their Mahara difficulties on their own or by asking fellow 
students, rather than with staff guidance. Because (left to themselves) only a small number of students 
in each group did the Mahara work, and this tended to be last minute, we expect this caused 
frustration and anxiety. To alleviate this, we recommend offering drop-in sessions in the run-up to the 


deadline and/or assessing draft work. 


Take-Aways 


Do induct students, ideally in person and in their groups, and ideally once they have begun to think 
about how to present their work. Do be available to support them in the run-up to the deadline. Do 
explain and repeat why you decided they should work in Mahara. Do make a range of examples of 
successful work available, and point students to a Mahara features demo (making one yourself will get 
you acquainted with Mahara). Please do make use of our guidance at 
https://wiki.ucl.ac.uk/x/DwXxAQ. 
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Chapter 8: Assessing Veterinary Students Using 
Posters and Online Lectures 


Sonya Powney?! 
Nick Short” 


Summary 


The eMedia Unit is responsible for delivering the ‘Digital Literacy for Vets’ module as part of the Masters 
in Veterinary Medicine (MVetMed) programme provided for graduate vets undertaking further training 
at the Royal Veterinary College. The course was developed to provide an overview of a wide range of 


new technologies, which could prove of value to vets in practice and academia. 


An authentic assessment method was sought to enable the students to put into practice the new skills 


learnt during the course. 


Implementation 


The course consists of eight face-to-face practical workshops, including the following topics: 
e Howto create high quality digital images 
e Working with digital video and audio 
e Using PowerPoint to create effective posters and presentations 


e Creating e-lectures and e-cases 


Students were asked to demonstrate these techniques by creating a scientific poster and an online 
lecture. Technical support and guidance was available during the scheduled sessions, and individual 


drop-in sessions were also offered. 


Assessment in 2014-15 and 2015-16 
e Production ofa scientific poster of clinical or research work. The poster will be marked on the 
basis of presentation, design and impact, but not on scientific content (50%). 
e Creation of a 10-minute e-lecture with narration and slide set on a clinical topic for use by 


undergraduate students (50%). 


Poster assessment criteria 


Based on the poster award criteria from the International Association for Medical Education (AMEE): 
e Attraction: Does it attract the viewer’s attention and is it visually appealing? 
e Design: Is it well presented and laid out to best display content? 
e Language: How well is the information presented? 
e Illustration: Are graphics, diagrams and images used effectively? 


e Message: How effective is the poster in projecting information and key conclusions? 


21 Case study author: Sonya Powney, Learning Resources Developer, eMedia Unit, Royal Veterinary College, University of London. 
22 Project lead: Nick Short, Head of eMedia Unit, Royal Veterinary College, University of London. 
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Evaluation 


Unfortunately, student participation in the formal student evaluation process was low and no feedback 
was received regarding the method of assessment. However, the quality of the student work was very 


high, and all the students who took part passed the module. 


Benefits and Challenges 


Benefits 


Posters and e-lectures provided an authentic assessment method, and encouraged students to reinforce 


their new knowledge in a practical way, hopefully better equipping them for the real world. 
Challenges 


The students came across some technical problems with the e-lecture recordings. Students were offered 
a choice of technologies, but they were unsure what to choose — in hindsight it might have been better 


to offer a clear recommendation (with alternatives available if required). 


It was very challenging to be objective when marking the poster and the e-lecture. The criteria for the 


poster helped with this to some extent. 


Take-Aways 


It is common knowledge that assessment drives learning. If practical real-world skills are assessed, 
students are more likely to engage and practise these skills, and, most importantly, will be better 


equipped for the workplace. 


This form of assessment helped students to embed the technical skills they had learnt. It also offered 


an opportunity for the course tutors to provide detailed feedback. 
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Chapter 9: Formative Assessment for Postgraduate 
Academic Skills Development in Arts 


Leo Havemann” 


Summary 


Step Up to Postgraduate Study in Arts is a taught, blended module designed to support students at the 
point of transition into postgraduate study (i.e. Master’s/level 7) across the various disciplines of Arts 
offered at Birkbeck. The module grew out of a Changing the Learning Landscape (CLL) project in 
2012/13, in which the Birkbeck project team” worked with CLL consultants” to develop a flexible, 
blended learning design approach suitable for Birkbeck students. Once this was developed, the next 


phase was the creation of a pilot module.”” 


Previously, academic skills training for potential or accepted postgraduates had been identified as a 
gap in provision, and also a potential route to support student retention. While acceptance on a 
postgraduate programme might be taken as an assurance of graduate-level academic capabilities, in 
practice many incoming Birkbeck postgraduates are returning to study after long breaks, and often 
will face significant work and family commitments. We asked programme leaders from the School of 
Arts to identify key academic capabilities that could be developed via a pre-sessional module. This 


wishlist evolved into the following learning units: 


Critical thinking 

Plagiarism, referencing and bibliography 
Resources for research 

Seminar skills 


The language of academic English 


oie! ee te 


Planning and structuring essays 


In terms of the mode of study, while the majority of the module is housed in Moodle, there are 
opening and closing face-to-face meetings which students are encouraged to attend. Participation in 
these meetings, as in all aspects of the module, is opt-in. Because Step Up is non-credit bearing, all 


assessment in the module is formative, although the final assessment tasks of each unit are graded. 


23 Leo Havemann, Learning Technologist, IT Services, Birkbeck. 


24 Leadership Foundation for Higher Education, Changing the Learning Landscape, https://www.Ifhe.ac.uk/en/research- 
resources/resource-hub/cll/. 


25 The original project team who worked with the CLL consultants comprised: Joana Barros, Lecturer in Geographic 
Information Science and former TEL Champion; Joanne Leal, Senior Lecturer in Cultures and Languages and Assistant 
Dean for Learning and Teaching, School of Arts; Liz Johnston Drew, Flexible Learning Tutor, School of Arts; Vincent 
Tong, former TEL Champion; Leo Havemann, Learning Technologist, IT Services. 


26 The consultants were Alison Le Cornu and Amanda Jefferies. 


27 The project team developed the pilot module, with additional content on academic English provided by Fleur Rothschild, 
Learning Development Tutor, School of Arts. 
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implementation 


Step Up utilises a blended, flexible approach in which the majority of the learning is self-paced and online. 
The six units in the online module space follow a consistent structure that opens with an introductory 
video and self-assessment quiz followed by a series of related learning materials. Each set of learning 
resources and activities is organised in progressive steps. These comprise readings, multimedia 
content, exercises, reflective writing, and tips for further learning. Finally, there is a marked 


assessment task. 


From a student’s perspective, much of the work of the module consists of formative assessment. The 
initial engagement with each unit is a self-assessment quiz which must be completed in order to ‘unlock’ 
the learning materials. The purpose of this self-evaluation is to enable students to assess their own 
knowledge and therefore plan their own journey through the units and assessments, prioritising the 
aspects they feel will be most helpful. Some of the activities in the learning materials are self-marking 
exercises, or consist of a reflection that will be responded to by a tutor (but not marked). The final 


assessment task in each unit is graded by a tutor who also provides individualised feedback. 


While almost all of the assessment and feedback in the module is technology-supported, one assessment 
in particular attempts to activate the transformative potential of a specific learning technology: Turnitin. 
Turnitin is used throughout the module, and throughout the programmes students will go on to study, 
but the nature of its use and functioning is typically in the background of academic activity. Students 
often do not have access to view their similarity reports, or place too much significance on the similarity 


percentage, so we felt they could benefit from a guided introduction to the similarity reporting function. 


The Turnitin-focused assessment closes the ‘plagiarism, referencing and bibliography’ unit and aims 
to surface and address academic writing expectations and anxieties. In this task, students are asked to 
write two summaries of an extract of a journal article. One summary should be deliberately plagiarised, 
and the other should avoid plagiarism. Both of these summaries are submitted via Turnitin and students 
are given access to both similarity reports. Contextual feedback is provided on each version using the 
Turnitin Feedback Studio environment (formerly, GradeMark), pointing out any errors or issues, and 
highlighting good practice. Overall feedback that considers how well they have addressed the task is also 
provided. Students are then asked to add a reflective post in their journals discussing their experience 
of completing the assessment. In this way, students are invited to explore the differences between 


similarity and plagiarism, and the nature of intentional and unintentional plagiarism. 


During this assessment activity, students observe that while Turnitin’s similarity reporting function 
can find matches to other known sources, it cannot draw conclusions about which of these matches 
are ‘significant’ in terms of plagiarism. It becomes evident that legitimate reuses of text can become 
part of the ‘noise’ that Turnitin reports generate, and also that it is possible to plagiarise an original 
source without generating any matches in Turnitin, simply by paraphrasing without citing. The core 
outcome of this exercise should therefore be an improved understanding of the importance of not 
reusing another author’s ideas without acknowledgement, whether one quotes or paraphrases, as 


academic inquiry is a process of ‘standing on the shoulders of giants’. 
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Benefits and Challenges 


The benefits and challenges of Step Up relate more to the module as a whole than the assessments 
specifically. The module has now run each summer for four consecutive years and is being used as a 
template for similar projects. Student feedback has been overwhelmingly positive, indicating that the 
content has been pitched at an appropriate level and has acted as a confidence boost as well as an 
academic refresher course. This has been confirmed by tutors who report that Step Up ‘graduates’ are 
entering their MA programmes already ‘up to speed’, participating in seminars with greater 


confidence and skill. 


The most significant challenge we face relates to the ongoing sustainability of the module. Both the 
original development phase and the subsequent delivery of the module have posed resourcing challenges. 
Although the online materials are reused from year to year, each instance of Step Up requires a 
significant commitment of staff time to moderate discussions, respond to reflections and mark the 
assessments, in addition to administration and the face-to-face events. As we prepare to run the 
module for the fifth time, we are increasingly seeing a need for revision and updating beyond 


maintenance and the smaller adjustments we have made from year to year. 


Another challenge is student engagement. Although all accepted postgraduates in the school are 
offered a place on the module, a smaller subset decide to take it up, and within that group there is a 
wide variation between those who engage enthusiastically in all aspects, and those who might log in 
and have a look or only do the self-assessments. While it has always been our intention that students 
should choose how much of the module they wish to engage with, it is difficult to be sure if we are 


reaching the audience which would benefit most from this intervention. 


Take-Aways 


Step Up has demonstrated the effectiveness of technology-assisted assessment practices in a flexible, 
blended learning environment. Self-marking activities developed in Moodle were combined with the 
use of learning technologies and individualised feedback from a tutor to model academic expectations 


at postgraduate level, so as to help students to identify and address gaps in their prior knowledge. 


More information on the flexible model and module are available as these were the focus of an innovative 
practice conference workshop.” The module has also been recognised by Jisc as an exemplar project” in 
the category of supporting students to study with digital technologies. The team were recipients of a 
Birkbeck Excellence in Teaching Award in 2016. 


28 http://eprints. bbk.ac.uk/8710 
29 https://digitalstudent. jiscinvolve.org/wp/exemplars 
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SECTION 3 


Case Studies — 


Students Feeding Back 


Chapter 10: Peer Reviewing Summative Assignments 


Sarah Sherman”? 
Sarah Channon?! 


Summary 


The first written assignment for veterinary medicine undergraduates in their first year causes much 
stress among the student body, as the stakes are high and, for many, this is their first experience of 
producing a piece of academic writing. For some students, the need for feedback and assistance with 
the assignment is especially great since they are used to getting extensive help - and in some instances 
coaching - at school. Unfortunately, since there are 200 students in the year group, it is simply not 


possible for students to receive the extensive one-to-one attention that they crave. 


The lead tutor needed to find a consistent approach that would reduce the load on staff time. She identified 
a technical solution to allow students to provide peer feedback within a specified framework of criteria. 
Firstly, the process of marking assignments different to ones they had submitted themselves gave the 
students access to a new topic. Secondly, the process of delivering feedback to their peers provided the 
students with the experience of putting themselves in the position of the tutor, and hence developing a 


new understanding of how to interpret assessment criteria in order to write a good assignment. 


implementation 


The ‘Workshop’ activity in Moodle provided an online space for students to upload their assignments 
while the tool was in ‘submission mode’. At a specified time, the tool changed to “assessment mode’, at 
which point no more submissions were accepted. Students were then manually (rather than 
automatically) allocated an assignment to review, so that they could be given a topic they had not 
chosen for their own essay. During the period of review, students completed a set of criteria 
statements which were added into the tool by the lead tutor. The statements mapped directly to the 
College’s common grading scheme. Once the feedback had been received, the students had a further 
week to submit their final version of the assignment. During this time, students were also able to 
submit a draft to Turnitin to generate a similarity report of their assessment. The peer review part of 


the assignment was not compulsory, but it was strongly expected. 


30 Case study author: Sarah Sherman, Service Manager, Bloomsbury Learning Environment. 


31 Project lead: Sarah Channon, Lecturer in Veterinary Anatomy, Royal Veterinary College, University of London. 
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Benefits and Challenges 
Benefits 


e The use of the tool enabled support for students that would otherwise not have been possible. 

e It was straightforward to set up. 

e ‘Workshop’ has configurable anonymity; in our case, removing anonymity allowed students 
to seek advice from each other. 

e Academic staff can also monitor and comment on the peer review. 

e The approach is helpful for students for whom English is their second language. 


e It encourages students to reflect on their work and how it maps to the marking scheme. 
Challenges 


Challenges to the approach stemmed from the students’ attitudes. For example, there was some doubt 
as to whether the tutor could be bothered to help them. There was also a tendency for students to 
mistrust one another’s opinions and reviews, so training students is recommended. These issues 


should reduce once students get used to the approach. 


Take-Aways 


If using the Workshop Tool in Moodle, it is really important to ensure that the deadlines are 
emphasised to students; once the tool has advanced from the submission mode to assessment mode 


(or assessment to feedback mode), it cannot be reverted. 
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Chapter 11: Using WebPA for Summative Peer 
Assessment 


Hannah Croall*? 
Bradley Cobb” 
Charlotte Lawson** 
Alistair Spark’ 


Summary 


The Problem Disease Investigation module in the RVC’s Biosciences Year 1 courses includes four 
weeks of group presentations; each week, the students peer-assess the other members of their group 


and these marks contribute to the overall summative marks for the module. 


In the past, the peer marks have been collected on paper sheets. However, it was felt electronic 


submission would be more efficient. 


Implementation 


The Course Support Manager, the Course Director and the Module Leader met with the RVC eMedia 
team to explain their requirements and discuss systems that could be used to collect peer marks. The 


VLE Manager set up a pilot electronic marking form using WebPA - http://webpa.ac.uk/. 
Accounts for all of the students were set up manually and students were emailed their login details manually. 


For each of the four weeks being assessed, an assessment form was set up and students were sent a 
weekly email prompt when the relevant form went ‘live’; students were then expected to log in and 


record a mark for every member of their group. 


We manually checked for outstanding marks and email reminders were sent to the students whose 
marks were missing. A deadline for completion of marks was set, and students were informed from 
the outset of the module that any student who did not provide peer assessment for their group 
members would not receive the marks awarded by their peers (meaning that their overall mark for the 


module would be lower). 


At the given deadline, we closed the mark forms and exported the peer marks. As peer marking was 
only one component of the whole module mark, we calculated an average mark for each student for 
each week; these marks were added to a separate spreadsheet used to collate the scores for all of the 


marked elements of the module. 


32 Case study author: Hannah Croall, Sciences Course Support Manager, RVC, University of London. 
33 Project team member: Bradley Cobb, Module Leader, RVC, University of London. 
34 Project team member: Charlotte Lawson, Course Director, RVC, University of London. 


35 Project team member: Alistair Spark, VLE Manager, RVC, University of London. 
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Benefits and Challenges 


Using WebPA enabled us to collate peer marks in a more efficient and confidential manner than the 
previous paper-based system had allowed. As the marks were a component of the overall summative 
mark for the module, it was important that they were collected accurately and securely. Another 


benefit was that the process became paperless. 


The students did not seem to encounter any issues using WebPA and the pilot went smoothly; we 


would be keen to use this system again in future years. 


As this was just a pilot scheme, many processes were completed manually (student log-ins were set 
and disseminated, students were assigned to groups manually, and email reminders/prompts were 
sent manually). It could have been easier if some of these processes were automated; however, the 

process worked fine and considering we were using a free system the work involved was not overly 


onerous. 
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Chapter 12: Peer Feedback with Moodie Workshop 


Mira Vogel’* 


Summary 


As part of a programme to support their application for Associate Fellowship of the Higher Education 
Academy, postgraduate teaching assistants undertake a peer feedback activity in three large groups. 
Each submits a case study of their own teaching practice, and provides feedback on three others, with 
reference to the UK Professional Standards Framework (PSF). The activity is feedback only, i.e. no 


numeric marks, and although non-compulsory, it is strongly encouraged. 


Implementation 


A Moodle Workshop was set up which gave a week to prepare the assessed work, and a week to give 
feedback on others’ work. We used Moodle Workshop’s Assessment Form to set three open questions 
which would scaffold the feedback by prompting reference to the UK PSF. Advance testing and evaluation 
of the first iteration gave us insights into what participants needed to know, to be reminded of, and how 
instructions would display to participants at each phase, and we prepared those accordingly. On the 
original submission we set attachments to zero (i.e. the submission would be typed/pasted into Moodle 
directly to streamline the review by allowing the feedback and original work to appear in the same 
window). We did not apply Moodle Groups to the Workshop activity since this would make the 
activity vulnerable to non-participation. We set the allocation to happen automatically, rather than to 
be done manually by a tutor. We ruled out non-submitting students from giving feedback on the basis 
that they were unlikely to participate. We were also very clear about who could see whose identity 
during the process (this is configurable using the Permissions for that instance of Moodle Workshop, 


and can be negotiated with students in advance). 


A tutor for each group introduced the activity during an in-person session. Since the participants were 
new to the process, the lead tutor in each group took the important measure of writing themselves diary 
reminders to move the activity along. This helped with a number of actions: to remember to click 
Close to reveal feedback; to monitor the activity at key points; to provide signposting for participants 
(e.g. how to find their allocated case studies, or how to access the feedback they had been given by 
peers); to send out emails flagging the deadlines. It should be noted that the submission deadline is 
‘hard’ for Moodle Workshop insofar as any lates need to be checked for and then manually allocated, 
which is time-consuming for the tutor, and potentially troublesome for any student who suddenly 


finds that they have an extra submission to assess which they had not planned for. 


36 Case study author and project lead: Mira Vogel, Digital Education Advisor, UCL. 
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Evaluation 


Twenty-one participants completed a short questionnaire. The response rate was low, but there was 
clear approval for the activity. Eighteen reported their experience of assessing peers’ work as Excellent 
or Good (with two further respondents reporting that it was Fine). Seventeen respondents reported 


their experience of receiving assessment as Excellent or Good (four as Fine). 


Benefits and Challenges 


Students said they appreciated: 
Feedback on their work. 
Insights about their own work from considering others’ work. 


Being able to edit their submission in advance of the deadline. 


The improved instructions letting them know what to do, when and where. 


Staff views haven’t been formally evaluated, but from informal conversations we know that the group 
leaders appreciate Moodle taking on the grunt work of allocation. However, this depends on setting a 
hard deadline with no late submissions (otherwise staff have to keep checking for late submissions and 
allocating those manually), and one of the leaders was less comfortable with this than the other. They 
found it feasible to write diary notes to send reminders and alerts to students to move the activity 


along. The exercise has been repeated and now includes all tutors. 


For each iteration we need to decide afresh about anonymity — should both submissions and reviews be 
anonymous, or one or the other, or neither? These can be configured via the Workshop’s Permissions, 
and it’s important to let students know who can see what. We also need to decide whether it is helpful 
for all participants to eventually be able to see all submissions and all feedback; we could achieve this 
by changing Permissions after the activity is complete (or even while it is running) to allow students to 
access the dashboard. We need to keep in mind that students who are reassured that their views are 
anonymous will often be franker than students who are not, and that their confidence to express views 


about peers’ work constructively under their own names should improve with time and support. 


Take-Aways 


Peer feedback in Moodle Workshop works well if it is closely monitored and guided until students are 
used to it. The first few times students participate will depend on a tutor monitoring the activity and 
intervening with reminders and the occasional click to move the activity along. However, tutors have 
said that this is an acceptable trade-off given the immense amount of time saved with the automatic 
allocation of work and return of feedback. Anonymity and visibility of work and feedback is highly 
configurable, which is another benefit. Students need to know if the submission deadline is hard, 
although with some effort on the tutor’s part it is possible to accommodate late submissions by 


making manual allocations of reviewers. 


60 


Chapter 13: Using Forums in Moodle to Provide Peer 
Feedback 


Sarah Sherman:’” 
J. Simon Rofe°*® 


Summary 


The purpose of the technique described here is to enable students to share professional and academic 
experiences, enabling them to learn, reflect on their learning and understand the purpose of assessment. 
It utilises the “e-tivity framework that identifies the following elements: purpose, task, response and 


outcome (Salmon, 2013). 


SOAS’s Masters in Global Diplomacy is a discursive course delivered online, with a variant available on- 
campus. The 200+ students come from a broad range of backgrounds and experiences. For each module 
(not including the dissertation), students have a specific time-frame in which to produce a 400-word 

analysis of an article, which they are required to post to a forum on Moodle. They are then required to 


provide an analytical review of 400 words on another student’s analysis under a specific rubric. 


This approach provides an opportunity for experiential learning, which is important in developing 
assessment literacy, and also provides exposure to additional subject-matter articles. The framework 
of assessment is a feature of all the six “e-tivities’ in each module. It is a scaffolded approach, which 


leads to a final essay submitted at the end of the module. 


Implementation 


This technique utilises the standard forum in Moodle, which is set up by academic staff. Students are 
arranged into groups of up to 15 (with a minimum of six) using the Moodle Groups function, and are 
assigned to an Associate Tutor. Students post their analyses of the articles to the discussion forum. 
They then choose a peer’s analysis to review, which is submitted to Turnitin as an assignment. The 
Associate Tutors mark the quality of the peer review as well as the analysis against a common rubric, 
ensuring consistency across the cohort. Each element is awarded a combined mark out of 5; GradeMark 


manages the marking. 


Students are introduced to this process at the very start of their module (within the first 72 hours), so 
the notion of a peer review is well understood and students get very used to communicating with each 


other using this approach. 


37 Case study author: Sarah Sherman, Service Manager, Bloomsbury Learning Environment. 


38 Project lead: J. Simon Rofe, Senior Lecturer in Diplomacy and International Studies, SOAS, University of London. 
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Evaluation 


The technique of peer feedback has produced engaged responses enabling deeper learning. The mode 
of assessment is included in the formal evaluation that is conducted at the end of each module by the 
Programme Director for informal reflection of its usage. Ultimately, it has been found that this is a 


low value, low risk method, which works well to achieve the learning outcome. 


Benefits and Challenges 


Benefits 


e Allows students to meet their learning outcomes. 

e Supports students’ engagement in Moodle. 

e Offers the opportunity for students to learn from each other. 

e Increases tutors’ confidence in using technology to support their pedagogy. 
e The technique works well for both distance and on-campus learning. 

e Engages academics without over-burdening them. 

e Easy and reliable technology — no issues have been experienced. 


e All managed in one place. 


Challenges 


The only issues experienced with the approach are administrative: if the academic setting up the 


assessment lacks the appropriate Moodle understanding and administration rights. 


Take-Aways 


Some simple training may be required to set this up. At SOAS, this is included in the e-moderator 


training programme that is offered to teaching staff by the Programme Director. 


References 


Salmon, G. (2013). E-tivities: The Key to Active Online Learning. Routledge, London 
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Chapter 14: Academic Blogging with Peer Feedback 


Lucia Guetcherian’’ 


Summary 


To promote deep learning in the Psychology of Criminal Behaviour, a module of the Criminology and 
Criminal Justice Programme (BSc and Cert. HE), the end-of-year exam was replaced with an academic 
blogging assignment. This assignment, in its four stages, is a process that imitates academic knowledge 
sharing and dissemination, which is a natural activity in academic circles. This mode of assessment goes 
beyond the traditional written essay or end-of-year exam. Academic blogging promotes and improves 
learner engagement and development through diagnostic and dialogic feedback. Learners will demonstrate 
transformative learning as they progress through the stages of the assignment, via research, collaboration, 
discussion in learner groups, drafting, peer feedback, and redrafting a revised edition of previous work 


following scrutiny by peers. 


The aim of this assignment is to enhance learner performance via diagnostic tutor feedback and peer 
information sharing in learner communities (groups). These inform learners about their progress and 
help to identify weaknesses and challenges that learners can address in order to construct new knowledge 
and improve their own performance. Using feed-forward and feedback techniques, students are assessed 
formatively throughout the assignment, except after final submission. The desired outcome is to enhance 
not only student confidence and competence in discipline-specific knowledge, but also to develop 


knowledge and skills that support employability. 


The main objective is to encourage learners to focus on understanding the subject matter critically and 


interpretively throughout the module and beyond the university campus. 


Implementation 


The assignment specification, uploaded to Moodle as a PDF document, included: questions, step-by- 
step guidance throughout the stages of the assignment, reflection questions and the marking criteria. 
The assignment requirements were also discussed with students in class in detail, and all questions 


were addressed. 


To demonstrate how to read peer output critically and provide constructive feedback, “Assessment 
Workshops’ were set up during seminars, where students were given the opportunity to scrutinise 
some scripts and examine the marking criteria, with guidance from the seminar leader, and apply 


them in tailor-designed classwork. 


Collaboration was loosely monitored, except when problems arose because of group dynamics or 
when students consistently failed to turn up to class. Every week, students were given opportunities to 


discuss the progress of their work or any challenges they faced. 


39 Lucia Guetcherian, Associate Lecturer in Criminology, Birkbeck, University of London. 
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At the end of the collaboration stage, draft blog posts were written and uploaded to an advanced 
Moodle forum in which the posts were anonymised (except those written by seminar tutors). The 
forum was set up eight weeks in advance of the deadline to manage stress levels resulting from 


unfamiliar territory. At this point peer feedback began. 


After two weeks of peer feedback and diagnostic tutor feedback (formative), students revised the blog 
posts, wrote a 1,000-word reflective essay detailing their experience on this assignment, created an 
appendix with feedback provided to peers, and submitted all three components as one document via 
Turnitin. The timeframe of two weeks for feedback turnaround and its dialogic nature throughout the 


two weeks kept the students engaged and focused on their tasks. 


Evaluation 


An analysis of the data indicates that learning in partnership with peers increased learner engagement and 
success. It worked to promote deep learning, as all group members were actively engaged in the process of 
learning itself rather than the outcome of their learning. It also contributed to forming learner communities 

and constructing a learner identity. On the other hand, through dialogic feedback, partnerships between 


the lecturer and the learner increased student engagement with their learning process. 


The reflective essay provided rich content for evaluation, where students had to provide feedback on their 
experience of the assignment. At first, students were quite apprehensive about this type of assignment, 
but, on completion, the vast majority of students enjoyed the different tasks very much and learned a 
great deal more than they would have done in an exam or, to a lesser degree, an essay. Learners addressed 
questions on the impact of collaboration, feedback, accessibility, mobility, deep learning, team support, 
best and worst features of the assignment, time management skills, technology navigations, independent 


research and employability skills. 


The results were very encouraging. Learning through collaboration and discussion encouraged 
students to articulate their ideas and questions, and to challenge and respond to questions and ideas 
from seminar tutors and/or peers. Group discussions helped students build up their discipline-specific 
knowledge and understanding, enabling them to consider new ways of thinking about and practising 
criminology. Feedback generated by the draft blog post helped learners to develop independent 
abilities to make judgements for themselves, cope with their studies and understand their weaknesses 
and strengths in order to address them before summative assessment. The option to use mobile 
technology to complete this assignment was very popular with learners, especially those with full-time 
jobs. They were able to access the material, provide and receive feedback and complete their 


assessment while commuting. 
To ensure that the language used in feedback was accessible to all, and all students and the lecturer 


had a common understanding of what was required in the assignment, assessment workshops were 


designed into seminar sessions. 
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Benefits and Challenges 


Collaboration and active engagement with the content of the course yielded excellent results in deep 


learning and understanding, and in constructing new knowledge, which learners disseminated (via the 


draft blog post) and on which they received peer feedback. 


Benefits (data gathered from reflective essays) 


Collaboration helped students improve their communication skills and helped them manage, 


set and achieve goals. 


Collaboration was enriched by the diversity of abilities and strengths per group, and 


weaknesses were addressed within the group. 


Giving and receiving feedback helped learners to examine one another’s work and to learn 
about different academic writing styles. Moreover, students said that tutor feedback was very 
useful, in particular that it contained details, explanations and suggestions on how the draft 


blog post could be enhanced. 


This assignment gave learners an insight into potential future jobs while developing and 


practising new writing skills. 


Having to work with others in a group encouraged learners to develop their critical thinking 


skills. 


The skills learned were transferable to other assignments/projects/modules. For instance, by 
learning how to provide feedback, students internalised the marking criteria, giving them a 
better understanding of the summative process, which in turn raised the standard of their 


output. 


Mobile technology, which allowed learners to complete the whole assignment via Moodle, was 


very popular among students. 


Excerpts from student feedback 


L 


“Blog writing is one of the fastest growing skills required in many places of work, but the 
assignment required collaboration and peer feedback which are crucial for being successful in 


a workplace environment.” 


“The blog requires a large amount of research, which is a core skill for many jobs and roles, 


but crucially it requires full understanding of the topic.” 


“By requiring it to be written with a public audience in mind, but still adhering to academic 


writing style, the assignment forced the writer to have fully understood the topic in question.” 
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Challenges (data gathered from reflective essays) 


e Inthe first iteration, some students did not receive constructive feedback from their peers. As 
a result, they had to rely only on lecturer feedback. This problem was addressed by embedding 
Assessment Workshops within the syllabus. 


e In the first iteration, learners worked in pairs, but in some cases learners dropped out of the 
programme without informing their peers. This resulted in high levels of stress. In further 
iterations, learners worked in groups of three or five members, depending on the size of the 
cohort. This worked well because students felt more supported by their peers and had better 


opportunities of collaborating and discussing their progress. 


e It took students a long time to understand what exactly the assignment entailed because they 
found the concept of collaboration and knowledge sharing alien to their previous modes of 
learning. An attitude change was fostered by addressing misconceptions during assessment 
workshops, where the rules of academic knowledge dissemination and importance of sharing 
knowledge were explained, highlighting that knowledge sharing is not plagiarism when 


academic conventions are adhered to. 


e Some students did not feel confident about navigating the stages of the assignment. In further 
iterations, to reduce stress, feed-forward sessions were embedded in assessment workshops 
and a list of ‘Frequently Asked Questions (FAQs)’ was provided to students. In the future, the 
FAQs will be uploaded to Moodle. 


e Formative tutor feedback at the draft blog post stage is very demanding, especially if the 
cohorts are large. 


Take-Aways 


1. Collaboration and peer feedback worked very well in developing student understanding of 
discipline-specific content and developing transferable skills. 

2. Assessment Workshops are essential to manage stress levels and maintain high levels of 
motivation and learner buy-in. 

3. The critical point is to ensure that student engagement with the feedback is transformative, i.e. 


they use the feedback as ‘feed-forward’ to inform their future work. 
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Chapter 15: Improving Feedback In Online 
Assessments 


Sonya Powney”’ 


Summary 


The purpose of this initiative was to improve communication between large cohorts of students working 


through online computer-aided learning quizzes and tutorials (CALs) and academic staff. 


In an e-learning student survey, conducted by the Royal Veterinary College (RVC) in 2014, students 
were asked what they thought was the ‘worst thing’ about online assessment and feedback. Some students 
commented that CALs were “impersonal - no place to ask questions about feedback” and “an excuse 
for a lecturer not to bother teaching”. The 2014 National Student Survey also highlighted general 
problems with feedback. 


The author noted the success of an interactive CAL session developed with Professor Adrian Boswood, 
where students were invited to ask questions at the end of each section of a Moodle quiz. These 
questions were then discussed in a face-to-face session. The author decided to try this technique with 


other RVC quizzes. 


Implementation 


The author contacted a handful of particularly enthusiastic academics to pilot this idea in the 


academic year 2014-15. They agreed to the addition of a new question towards the end of the quiz: 


“Do you have any questions about the topics covered in this CAL? Are there some aspects you still find 


difficult to understand? Your questions will be collated and answered via an email to your year group.” 


(This was followed by an existing question asking for feedback about the CAL in general, asking what students 


liked about it, and how it could be improved, to spot any mistakes and to generate ideas for improvement.) 


A week after the end of the timetabled session (as students rarely worked through the CAL when 
scheduled), the author collated the student questions and emailed them to the relevant academic, who 
had the choice of whether to present feedback via an email or during a face-to-face session. Most 


opted for an email to the relevant student year group. 


Feedback from staff and students was overwhelmingly positive, and therefore the approach was rolled 
out to all RVC Moodle CAL sessions in 2015-16. 


Evaluation 


An informal evaluation took place. The author emailed the academics who took part in the pilot 


phase, asking for feedback: 


40 Case study author and project lead: Sonya Powney, Learning Resources Developer, eMedia Unit, Royal Veterinary 
College, University of London. 
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e “Very useful.” 

e “Very valuable.” 

e “This made the process of feedback much easier than individual students emailing me - I find 
that some of the questions are repetitive anyway.” 

e “Interesting to see the comments the students have made.” 

e “Interesting to see how students perceived the CAL. Will tweak it a little bit next year to make 
sure that they do not spend too much time on it and do not get lost in details.” 


e “Lots of ideas how we can improve things next year!” 


The students were also asked for feedback (via the inclusion of yet another question in a scheduled 
CAL session). The response was entirely positive — here is a selection of comments: 

e “This is very useful. I don't always have questions, but sometimes other people ask questions 
which I should have thought of and am interested in finding out the answer for! And it helps 
to engage with the lecturers too.” 

e “TI feel as though it has allowed the lecturers to get a feel of how we have grasped the concepts.” 

e “This is useful as it helps to clarify any points that were unclear and you get to see answers to 
other queries you may not have thought of.” 

e “Ihave found this very useful as, in the past, if you didn't understand something about a CAL, 
it was very difficult to get feedback that totally answered your question.” 

e “You can ask questions more anonymously, whereas before you may have felt too 
shy/embarrassed to ask something that may be seen as a ‘silly question’.” 


e “Please keep giving us the year feedback, it’s really helpful.” 


Benefits and Challenges 


Benefits 


Students were given the opportunity to anonymously ask questions about topics they found difficult 
or confusing, whereas in previous years they might have struggled alone. Academics could easily 
respond to the queries, sending their response to the whole year group, so other students could learn 


from the exchange. 


Challenges 


A small number of academics were reluctant to engage - concerned that it would increase their busy 
workload. They were reassured that there were normally only between 5 and 15 questions to respond 
to, and that they simply needed to add comments onto an email sent to them and forward it to the 
relevant year group. A few members of staff needed some prompting to respond to the students, but 


only one or two failed to provide feedback as requested. 


Take-Aways 


This is a very simple and easy-to-implement approach. As well as making academics aware of student 
areas of difficulty, and enabling them to quickly respond to them, it makes students in large cohorts 


feel more in touch with teaching staff. 
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SECTION 4 


Case Studies — 


Assessing at Scale 


Chapter 16: Assessing at Scale in a Global Health 
MOOoC 


Joanna Stroud“! 


Summary 


LSHTM delivered its first massive open online course (MOOC) in the field of global health in January 
2015. Each free online course, delivered via the FutureLearn platform, has made use of a range of content 


delivery methods and learning activities. 


With far greater numbers of learners than we are used to in a traditional context, it is important that 
the pedagogies employed are fit for purpose at a ‘massive’ scale and effective without tutor intervention; 
this is particularly true for assessment tasks. As such, it has been key to select methods that can mirror 
the course or educator voice, while being automated, and offer supportive feedback that motivates the 
learner towards the appropriate solution or outcome, at the same time as being rigorous enough to 


challenge learners’ understanding and application of material. 


Multiple choice questions, with either single or multiple answers, and structured, pre-defined feedback can 
perform in such a way, and have therefore acted as the fundamental form of assessment throughout 


our courses. 


Implementation 


Simple quiz activities have been employed at appropriate points throughout each free online course, 
and are targeted towards assessment of learners’ understanding of theoretical concepts rather than 
current or historical issues. Quiz questions require that students apply the concepts and understanding 
they have been exposed to, rather than demonstrate their retention. Every question also incorporates 
general and per option feedback that, in the case of an incorrect answer, implicitly and in a meaningful 
way signposts learners towards course material that is of use in seeking the correct answer. In the case 
of a correct answer, the feedback received is both targeted and formative, working to underscore the 
reasoning for the option to be correct and, where possible, extending knowledge through access to 


advanced material. 


While these assessment activities are ultimately low stakes given that there are unlimited opportunities 
to yield the correct answers, an individual’s attempt and score do count towards the ability to 
purchase a Statement of Participation (attempt required only) or Certificate of Achievement (pass 
mark of >50%). 


41 Case study author and project lead: Joanna Stroud, Distance Learning Facilitator, UCL; at the time of writing the case study, 
she was E-Learning Officer, London School of Hygiene & Tropical Medicine. 
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Benefits and Challenges 


Multiple choice quizzes are a valid and worthwhile form of assessing learners at scale. They can address 
gaps in learners’ understanding quickly and efficiently, and provide instantaneous feedback. However, 
a challenge common in the design of multiple choice questions in any context is to constructively align 
each quiz task to the teaching week’s learning outcomes, and to instil effective question design techniques 
that target higher order thinking rather than recall. Nonetheless, for the staff involved this challenge 
became a benefit, with the training given in this area being applicable and transferable to their regular 


teaching and assessment practices. 


Take-Aways 


A valuable paper with respect to effective use of multiple choice questioning is David Nicol’s ‘E-Assessment 
by design: using multiple-choice tests to good effect’: 


http://www. reap.ac.uk/reap/public/Papers/MCQ_paperDN. pdf. 


Another short blog post from the University of Sussex also targets quiz design in a practical fashion: 
http://blogs.sussex.ac.uk/tel/2015/08/04/effective-online-quiz-design/. 
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Chapter 17: Assessment in MOOCs for Continuing 
Professional Development 


Tim Neumann” 


Summary 


UCL Institute of Education’s first MOOC ran on the Coursera platform for six weeks between May 
and July 2014 with an intention to provide Continuing Professional Development (CPD) for Primary 
Education Teachers based on a co-learning strategy. The pedagogy of this CPD MOOC followed the 


standard CPD format of curating resources and orchestrating peer collaboration. 


The MOOC was designed and delivered by a team of eight international authors from four continents 
under the leadership of Professor Diana Laurillard, with additional support from a small team of teaching 
assistants. Learning objectives were centred around sharing of experience and practice, to enable 
participants to develop and improve strategies for Information and Communications Technology (ICT) 
implementation in schools, to support head teachers and specialist staff in developing pedagogy-led and 


problem-led uses of ICT, and to gain greater awareness of the range of pedagogic innovation that uses ICT. 


Such learning objectives and the CPD nature of the course are not in line with the provided assessment 
options on a typical MOOC platform, which is centred around quantitative methods and automatic 
grading using quizzes or formulas with easily identifiable ‘correct’ solutions. The design team worked 
creatively to bridge the expectations from the MOOC provider, the pedagogic experts and the 
practitioners. The adopted solution focused primarily on peer assessment, but also on the quantitative 


aspects of forum contributions. 


Implementation 


In line with standard CPD design, there was no formal assessment. The automated assessment 
methods using quizzes were not appropriate for evaluating qualitative descriptions and localised 
interpretations of theory that were the main outputs from participants. Peer assessment was the only 


form of assessment used. 


Each assignment had a set of criteria, and once a participant had submitted their assignment, these 
were sent to up to four peers to assess against the criteria, giving marks out of ten and constructive 


qualitative feedback. 
The four peer-assessed assignments were to: 
e Create and describe a learning object 
e Develop a ‘technology decision’ for their school or class 
e Prepare a suggestion for tackling one of the key challenges identified in the literature 


e Design a two-page brochure or newsletter to inform policymakers about the issues of implementation 


42 Case study author and project lead: Tim Neumann, Learning Technologies Lead, UCL Knowledge Lab, UCL Institute of Education. 
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In each case participants could use what they had produced during prior activities on the course as part 
of their assignment. The requirement to review the outputs of their peers was also an opportunity for 


participants to use this experience to improve their own outputs. 


This peer assessment represents instances of reciprocal learning activities, but it is not collaboration, 
which was the main pedagogic aim. Collaboration is only possible within small groups, but the MOOC 
platform that was used did not have a suitable group function at the time. Typical collaborative tasks 
such as co-working on a shared output is currently difficult to achieve on a MOOC platform, and even 


harder to assess. 
As the MOOC platform expects some form of judgement on whether a course has been completed 
successfully, pass and distinction levels were defined: 

e Pass level: 60/100 


e Distinction threshold: 85/100 
The final participation points were calculated from: 


e Number of submitted peer review contributions (20% for 4 in total in weeks 3-6) 
e Number of peer reviews done (30% for 8 in total) 
e Peer feedback scores (30%, based on scores for each of 4) 


e Forum contributions across the course (20% for 15 in total) 


Only those who did all four assignments were able to achieve a distinction. Average scores for weeks 
3-6 were 68%, 75%, 79% and 79%, respectively. The slightly weaker participants were dropping out, 


perhaps, but the range is not very wide. 


Of those active in week 6, 27% gained a distinction and an additional 10% gained a pass, both towards 
the high end of MOOCs delivered via ULIP, of which this MOOC was a part. 


Evaluation 


The MOOC was evaluated in detail, with a post-course participant survey, participation analysis based 
on activity statistics (video views, forum views and posts, and similar), marking statistics, forum comment 


analysis, and additional analysis of wider data (mainly for marketing success measurement purposes). 


Most of the evaluation focused on target audience reach and general experience with the MOOC. In 
the absence of formal assessment, success in relation to the learning outcomes was measured by evaluating 


participant expectations, perceived improvements and course experience. 


Respondents valued the overall course experience highly with 99% of responses in the ‘excellent’, ‘very 
good’ or ‘good’ category. Based on a pre- and post-course survey comparison, their perceived 
improvement in subject understanding developed significantly, and a clear majority of respondents 
with over 90% of strong or moderate agreements indicated that the course was relevant to their career, 


that own goals were fulfilled, that expectations were met, and that the time investment resulted in 
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satisfactory learning outcomes. With a response rate of just above 20% of active final week users, these 
results are not fully representative, but they show that even without formal assessment, it is possible to 


evaluate the success of a course in terms of learning objectives and outcomes. 


Interaction patterns in this specific MOOC deviated significantly from comparable MOOCs, in that forum 
discussions were much higher, with almost 40% of active final week users posting messages, as opposed 
to 2-3% in other MOOCs. While the level of interaction itself can be regarded as evidence for the 
existence of an active learning community, it is not a measure for achieving learning outcomes. 
Nevertheless, interaction in optional activities was about 20% lower than for compulsory activities, so 
making activities compulsory leads at least to higher engagement. How this translates to learning would 
be a matter for a deeper forum comment analysis, which is not doable at this scale. The comment analysis 
done for this evaluation uncovered evidence of co-learning, which again points to a success in meeting 
the objective of creating a learning community, but this data cannot easily be used to assess participants 


at an individual level. 


The peer assessments were a highly useful driver for direct interactions between participants. The overall 
grade was effectively a peer validation of whether the contributions reached a satisfactory level, but the 
real value was in writing responses. “Doing the peer review was the highest-rated activity, with just under 


85% strong or moderate agreements, topping course videos and discussion forums. 


Benefits and Challenges 


MOOC platforms guide course designers towards using standardised quantitative assessment that might 
not be appropriate for all purposes. Our MOOC demonstrates that the platform we used can be used 
successfully for co-learning with formative peer assessment, while still providing the numerical data 


used by the MOOC platforms to decide on whether an individual has completed a course successfully. 


The pedagogic design, however, needs to be developed carefully, and some creativity is required to 
configure the functionality in a way that allows learners to work effectively towards CPD learning 

objectives, while providing the assessment data format required by the MOOC platform. In essence, 

participation levels can be monitored automatically by the platform, whereas content assessment is 


done by peers. 


While the MOOC platform for our MOOC did not provide appropriate functionality to facilitate proper 
collaborative learning, most notably mechanisms to form and guide small groups at scale, the functions 
were sufficient to enable a good level of co-working for the benefit of participants, although tutor 


intervention is likely to be needed for such an approach. 


The wide reach of MOOC platforms and their ability to draw in high numbers of participants make them 
an attractive delivery mechanism for CPD. Provided that future platform policies will not shut the door 
to working around largely automated assessment methods based on quizzes or formulas, and provided 
that platforms will allow for pedagogy-driven appropriate strategies for formatively assessing co-created 
outputs, qualitative descriptions and localised interpretations of theory, CPD can run successfully at 


scale on these platforms. 
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Take-Aways 


Learning points, desired functionality for MOOC platforms and details on the MOOC, including the 


assessment methods, can be found in the following open publications: 


Laurillard, D. (2014). Anatomy of a MOOC for teacher CPD. UCL Institute of Education, University 
College London. Available: 
http://www .iite.unesco.org/files/news/639194/Anatomy_of_a_MOOC.pdf 


Laurillard, D. (2016). The educational problem that MOOCs could solve: Professional development 
for teachers of disadvantaged students. Research in Learning Technology, 24. Available: 


https://journal.alt.ac.uk/index.php/rlt/article/view/1738 
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Chapter 18: Self-Assessment and Self-Monitoring 
Tools in Professional Accountancy 


Stephen Ogden* 


Summary 


We are developing a new range of online programmes that are intended to be delivered to large cohorts. 


It is essential that we are able to provide high quality support and feedback in a scalable manner. 


With this as a driver, we sought to create and provide a range of self-assessment and self-monitoring 


tools for students. 


This case study describes three complementary tools we implemented to cover self-assessment, self- 


reflection, progression analysis and understanding, mapped against learning outcomes. 


Implementation 


Study content is divided into topics; each topic has a set of objectives, which map to overall learning outcomes. 


Three tools were implemented. 


Self-mark questions: self-assessment, self-reflection 
e Each topic has several quizzes which contain approximately five questions. These questions 
come in two parts. 
e Each question requires a long-form answer which might take 5-10 minutes to write. 
e After all questions are answered and submitted, a set of detailed model answers are displayed. 


e Students are then prompted to grade their own answers against the model answer with a scale 
of 1-5. 


e These quizzes test knowledge to a greater depth than multiple choice questions and do not 


require tutor intervention to mark. 


e Additionally, and importantly, these questions then push students to engage in reflective 
thinking to self-assess their understanding and the answers they provided. 


e Each question contains metadata mapping them to learning outcomes. 


43 Case study author and project lead: Stephen Ogden, Head of Learning Technology, University of London International 
Academy (UoLIA). 
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Topic objective tracker (TOT): self-reflection, progression analysis 
e At the end of every topic of study, students are prompted to fill in their progress log (a self- 
diagnostic form). 


e This allows students to reflect and self-assess their confidence and understand each topic 


outcome ona scale of ‘Completely confident’, ‘Partially confident’ and ‘Unsure’. 
e Each progress log collates into a single TOT, which students can use to monitor their 
understanding of each topic across the whole module of study. 
Learning outcomes profiler (LOP): progression analysis, understanding 


e Each topic contains self-mark quizzes, as well as some conventional multiple choice quizzes. 


All questions (in both types of quiz) are mapped against learning outcomes. 


e The LOP collates the results of all quiz questions, mapping them visually against learning 


outcomes. 
e For each learning outcome, the student receives two scores. 


e The first score correlates to the number of questions they have attempted (i.e. attempted five 
questions for a given learning outcome against a total of ten). This indicates their progress 


through the assessment content. 


e The second score correlates with their results: the number of correct answers they have 
achieved taken from the total number they have attempted. This indicates their understanding 


of the subject matter. 


Evaluation 


Weare less than one year into using these tools so are not in a position yet to formally assess their success. 


Weare pleased with the anecdotally large volume of students using these tools, and with the high coursework 
submission rates and completion rates on the new programme implementing these tools as compared 
with programmes not using these tools. It must be noted, however, that this new programme does use 


a range of additional approaches to encourage participation, attainment and retention. 


We will be asking students for their feedback on the utility of these tools via surveys. 


Benefits and Challenges 


There are, hopefully, obvious benefits to giving students greater ability to monitor and take responsibility 


for their own progress. 


One significant additional benefit is the quantity of data these tools generate in an easily comparable 
format. Although still work-in-progress, this allows support staff to monitor overall progress of large 
cohorts and the data may be used to identify students who are struggling or lacking in confidence. 
Where possible, the data can be used to generate directed interventions with the goal of increasing 


student attainment and retention. 
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One aspect we are currently missing is linking the topic objective trackers to the learning outcome 
profiler. We are very interested in correlating how well the student thinks they understand the content 
with how well they actually perform in quizzes. All the relevant information is available and this 


linking can be performed manually, but automating this would be very valuable. 


Another area in active development is extending the functionality of the LOP to provide a level of 
adaptive assessment. The tool already indicates attainment against learning outcomes and all quiz 
questions are already mapped against these. The next step is to allow the LOP to generate new quizzes 
containing questions mapped against learning outcomes which the student is struggling with and/or 


selected learning outcomes. 


Take-Aways 


All of our self-assessment tools are optional. While we believe they have a great utility to students, we 
are not in a position to mandate their use. This means that some generated data indicating a lack of 


participation does not automatically correlate with a student struggling to understand. 


Future iterations will attempt to improve usage by highlighting to the students the benefits of using 


the tools to make their own study more efficient, and potentially improve their assessment scores. 
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SECTION 5 


Case Studies — 


Multimedia Approaches 


Chapter 19: Student Presentations at a Distance 


Tim Neumann“ 


Summary 


Key academic skills are the presentation of research and presenting critical, reflected arguments. For 
distance education students, the predominant mode of communication is text, and there might be limited 
opportunities to talk and engage in immediate interactions. While the technology exists to connect 
students by voice, by video and generally by types of audiovisual media in real time, such sessions are 


often still tutor driven, and sometimes even restrict students to textual reactions. 


At the UCL Institute of Education, several modules run activities that ask distance education students to 
make individual or group presentations in webinar sessions, aiming to develop their academic skills and 
to engage them in formats of discussions that are common academic practice. While those activities 
currently are not assessed summatively beyond a participation requirement, formative group feedback 


is provided to support the overall development of academic skills. 


This case study addresses two types of student presentation activities that are used in three postgraduate 


modules, along with observations on peer and tutor feedback. 


Implementation 


Two activity types will be described, with a focus on the feedback aspects. Both activities are run as 
webinars using the Blackboard Collaborate platform, complemented with two text-based 
asynchronous forums, one for preparatory discussions and one for post-event discussions. The forums 
mainly exist as a peer feedback resource, while tutors restrict their responses to clarifications of the 


presentation process, expectations and practical problems. 


Activity type: research presentation 


Students are asked to prepare a presentation of their research or a piece of research they find interesting. 
Presentations are of fixed duration and must be supported by a set of slides. To accommodate the 


widest range of options, students are given the choice between: 


1. A 10-minute live presentation plus 5 minutes for audience questions. 
2. A pre-recorded video of 10 minutes, plus either 5 minutes of questions during a webinar 
session or asynchronous text questions if they cannot make any webinar session. 
A 6:40 PechaKucha® as a webinar presentation or a pre-recorded video plus 8 minutes of questions. 
4. A presentation, either face-to-face or video-streamed, during an annual IOE student 


conference event. 


44 Case study author and project lead: Tim Neumann, Learning Technologies Lead, UCL Knowledge Lab, UCL Institute of Education. 


45 A 6:40 PechaKucha is a presentation delivered at speed where 20 slides are presented for 20 seconds each, lasting 6 
minutes and 40 seconds in total. 
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A high number of two-hour webinar slots at a wide variety of times are offered for student booking, with 
up to six student presentations per session, to allow for slippage and reschedules. The number of webinar 


sessions overall therefore depends on the number of students. A tutor acts as chair of the proceedings. 


Feedback is provided as a group discussion at the end of each session, based on notes taken by the 
tutor during presentations and organised under main headings. Students are encouraged to be active 
participants in the feedback discussion, thus providing opportunities for peer feedback and immediate 


responses to feedback. 
Activity type: debate 


Students are given a ‘house opinion’ to debate. They are allocated to two groups by the tutor by random 
allocation, in order to force some to argue against their instinctive preference, leaving their comfort 
zone. Groups are then given access to a private discussion forum and a private webinar environment 


for debate preparation. 


The debate itself runs as a live webinar session scheduled for a maximum of two hours to account for 
slippage and feedback, and it is organised into four rounds with an eight-minute single-presenter slot 
for each round. Students are free to use visual aids such as slides; they are asked to make their points and 
engage with the opposing side’s arguments. During the live webinar, students are free to communicate 
in private in the background to adjust their arguments in response to the opposing side’s presentation. 
During a presentation, students are allowed to use the webinar’s text chat function in response to the 


presentation, although the presenter can choose whether to respond to or ignore the text chat. 


During the debate, the tutor acts as chair of the proceedings. At the end of the debate webinar, the 


tutor provides formative feedback and invites students to respond in a discussion. 


Votes are held before and after the debate to establish the extent to which the debate has changed 


opinions; a discussion forum is then provided to pick up on points raised in the debate. 


The webinar date and time is agreed by putting a number of different time slots to a vote. 


Evaluation 


The activities were continuously monitored as part of the normal end-of-module evaluation processes. 
In addition, anecdotal statements were collected from student forums, and usage statistics and activity 
engagement data from Moodle and Blackboard were monitored. Anecdotal statements might display a 


positive bias, as forums appear to be used less for critical responses. 


The statistics generally showed a very good level of pre-presentation engagement. Students who contributed 
less to the pre-presentation forums and also did not take part in webinars tended to participate less in 


post-presentation forums, highlighting again that the captured statements might be positively biased. 


The statements themselves were generally highly positive about both types of activities. Students appreciated 
the opportunity to listen to each other and debate with each other in real time, even though the number 
of actual speakers in a session was limited. Engagement by listening and using the text chat, or even 
simply by listening, enhances the social cohesion significantly and appears to provide an additional 


level of motivation to students. 
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Students did, however, comment on a relatively high time investment for both activities. While some 
concerns echoed a regret that the activity was not part of the overall summative assessment, there was 
a general consensus that - despite initial doubts by some - both activities were positive learning 
experiences, and students seemed to be able to identify learning gains very directly. The debate activity 
showed particularly high engagement rates, with a display of genuine excitement, manifest in frantic 
background activity, and a high number of unsolicited comments that rated the debate as the 
highlight of the module. 


The high tutor visibility during this activity also appeared to compensate somewhat for a lack of 
presence during other phases of the module. The group discussion format of the end-of-webinar 
feedback session was also widely accepted positively, which contrasts with requests for individual 
feedback for other activities. Tutor feedback is an important part in distance education, and the 
encouraging comments about live group feedback discussions might be a good complementary 
method of keeping the provision of formative feedback manageable in terms of workload in groups 


with a high student-to-tutor ratio. 


Benefits and Challenges 


Student presentations and live debates offer the potential for high levels of deep student engagement. 
Tutor feedback can be organised into group feedback discussions, which appear to be accepted positively 


by, and genuinely useful for, students while not being overbearing for tutors. 


Time differences are an obvious challenge for any live interaction in a distance learning context, with 
students spread all over the world. Tutors should be willing to offer sessions outside of traditional office 
hours to accommodate as many students as possible. Where this is not possible, for example because 
only a single session is run as in the case of the debate activity, opportunities for engagement must be 
created for those students who cannot attend the scheduled sessions. Pre-recording of presentations is 


one such opportunity, while forums for preparation and debriefing are another. 


Supporting pre-recording of videos might be a challenge; a good set of instructions or pointers to self- 
help resources are beneficial. Occasional difficulties might arise when students have technical difficulties 
with their microphone during a webinar or difficulties connecting altogether. Not all issues can be 
resolved on the spot. In the case of individual presentations, the tutor can ask students to prepare 
presentation notes, which the tutor can read out alongside the slides if there are unforeseen issues with 


a microphone. 


Take-Aways 


The activities should be embedded deeply into a module curriculum. While we have been running 
these activities at Postgraduate (PG) level, they would be suitable for Undergraduate (UG), too, 


though students might need more guidance. 
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The activities require some logistical effort on the part of the tutor. The process of finding webinar 
dates in particular should be initiated as soon as possible, with clear deadlines for session registration. 


The absolute minimum lead time is two weeks. 


Individual student presentations require a high level of tutor availability during presentation week, 
though this is no different to equivalent face-to-face activities. The debate activity, once times have 


been agreed, runs pretty much by itself before and after the webinar. 


Feedback categories should be worked out well in advance, and tutors must be very familiar with 
them. There are arguments for and against releasing the categories in advance, so this should be up to 
the tutor. The webinars themselves require a high level of concentration in order to generate useful 
feedback points, but providing feedback as a group session opens up highly beneficial opportunities to 


engage with feedback in more detail. 


Time management during webinars must be strict, otherwise slippage will be significant. 


Tutors should emphasise character of these activities, as well as the very tangible benefits of working 
together — in the event of individual presentations, students would still work together on providing 


peer feedback. 
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Chapter 20: Using Online Communication Tools to 
Provide Live Feedback to Students 


Sarah Sherman“ 
Sophie Pullen” 


Summary 


Providing essay support to students both on distance learning courses and during the holiday periods 
can be challenging. Traditionally, students receive help and advice in written format, either as comments 
on a Word document (often using the track changes option) or using the various functions available in 
Turnitin. Feedback in this form takes time to produce and is often unsuccessful. Many students have 
reported that they find written feedback confusing, and some find the amount that has been written 


demoralising. This seems particularly true in the case of students with learning difficulties. 


In order to address these problems, students at the RVC are now offered essay support via Skype. This 
has proved to be popular, as it provides students with the opportunity to discuss areas of concern and 
obtain clarification on points they do not understand. It is also far more satisfying for staff, as it enables 
them to see each student as a person and not just a piece of work. The conversations between student 
and staff allow both parties to identify the issues together, discuss ways of addressing the problems 
together and develop an action plan. It is less time-consuming to support students using this method, 


and more enjoyable and more beneficial to the student. 


implementation 


Students who are struggling to write an essay, have received poor feedback on a draft submission or who 
are required to resubmit come to the RVC’s Learning Development office displaying a variety of emotions. 
Students often feel ashamed at their inability to write, embarrassed by the results they obtained, frustrated 
by their lack of understanding, angry at the process, and many have lost complete confidence in their 
abilities. Providing support to these students is challenging and adding to their work using more written 


words, often interpreted as criticism, is often contra-indicated. 


In order to ensure that the Skype feedback session is constructive, each student is asked to email their 
essay and, if applicable, a copy of the feedback they have received from their tutor. The essay is then 
read and key issues that need to be discussed are highlighted. For example, problems with referencing 
may be highlighted in green, changes of topic within a paragraph may be highlighted in yellow, etc. 
Occasionally, a short sentence (in a different colour) is added to explain a situation or provide an example. 
Informal language is used, writing is kept to a minimum and emoticons are often added to try to 
reduce the student’s anxiety. The ‘colourful essay’ is then emailed back to the student within 24 hours 


of the Skype appointment. 


46 Case study author: Sarah Sherman, Service Manager, Bloomsbury Learning Environment. 


47 Project lead: Sophie Pullen, Learning Development Officer, Royal Veterinary College, University of London. 
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During the Skype session, the essay is discussed by colour, allowing both parties to navigate through 
the essay with ease, identifying the points which need to be discussed. Students are asked why they feel 
certain words/phrases have been highlighted and are encouraged to find ways of rectifying the problems. 
Students are then asked to look at other paragraphs within their essay, which have not been highlighted 
(a change in font colour allows these paragraphs to be identified), to see if they can identify similar issues 
to those that have been discussed previously. Skype enables this conversation to take place and allows 
students to develop ways of identifying their own problems and addressing their own mistakes. 
Similar sessions can be, and have been, conducted over the phone; however, experience suggests that 


these conversations are often more one-sided and more formal. 


Evaluation 


No formal evaluation has taken place; however, anecdotal evidence suggests that students who use Skype 
for feedback, or who are willing to discuss an essay in the same way over the phone, find the process 
useful. The following quote is from a student on a distance learning programme, and demonstrates 
the benefits of discussing essays and encouraging students to identify problems so that they can apply 
what they have learnt to other pieces of work. This process ultimately helps the student and reduces 


the amount of support they require: 


“Just would like to say a massive thank you for your help and guidance with my summative 
EPCRs for my ECC module before Christmas. I got 75% on the one you went through with 
me and 68% on my second one. I am extremely happy with my results and hopefully I can 


continue writing at that level” (Graduate Diploma Student, 2016). 


Benefits and Challenges 


Benefits 


e Students are able to record the conversation and play it back at a later date. This is particularly 


useful for students who may struggle to converse, listen and write at the same time. 


e Itis afar more enjoyable and constructive use of time for tutors. Less time is spent writing 
comments, knowing that they may be misinterpreted and/or not read, and more time is spent 
encouraging the student to identify their own problems and find their own solutions, which in 


turn increases their confidence. 


e The use of Skype improves conversation as body language can also be conveyed. 


Challenges 


e Scheduling the Skype call can be problematic due to student placements - the calls often take 


place at lunch times, late in the afternoon or evenings. 


e Poor Internet connections can occasionally be a problem from some remote locations. 
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Take-Aways 


This method of providing support needs to be promoted more to students on distance learning 
courses. We have not done this effectively in the past, probably due to a fear of Skype. However, 
having witnessed and experienced the benefits and noticed that issues with Skype are not as bad as 


initially anticipated, we will be promoting it more in the future. 
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Chapter 21: Providing Audio Feedback to SOAS 
Chinese Competition Contestants 


Sultan Wadud* 
Zhaoxia Pang” 


Summary 


For the first time at SOAS, 20 third-year undergraduate students were being entered into an annual UK 
contest on Chinese proficiency. The course leader wanted to provide students with feedback in preparation; 
however, it was felt traditional methods such as written feedback were not effective in conveying accurate 
pronunciation, and individual face-to-face meetings were deemed too time-consuming. This case study 
describes how the course leader was able to provide individually-recorded feedback to students for 


verbal assessments through the use of simple technology. 


implementation 


The course leader used a headset with a mic plugged into the office computer to record the short (few 
minutes) individual feedback for each student using free software, Audacity. The recordings, which were 


in mp3 format, were then emailed individually to each student. 


Students were informed in the face-to-face sessions about how their feedback would be delivered, so 


they were fully aware of when and how they would receive their feedback. 


No issues were reported by the students of any technical problems encountered in accessing the feedback. 


Evaluation 


The SOAS students performed exceptionally well in the competition, and won the contest to become 
no. 1 in the UK. The response provided by the students to the approach was positive; in particular, 


they liked being able to access the feedback on their mobile devices. 


Benefits and Challenges 
Benefits 


Students could access their feedback any time, any place and on their preferred devices, thereby offering 


flexibility over the traditional method of receiving face-to-face feedback. 


They were also able to re-listen to the recordings as many times as they wanted. 


Challenges 


There were no major challenges, just the time required to record and send out the recordings to the students. 


48 Case study author: Sultan Wadud, E-Learning Officer, SOAS, University of London. 
49 Project lead: Zhaoxia Pang, Senior Lector in Chinese, SOAS, University of London. 
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Take-Aways 


This type of assessment feedback is very effective for small classes, but may become difficult to manage 
alone with a larger cohort. 


It is vital that students are informed properly at the start of the course as to how the feedback will be provided. 
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Chapter 22: Moving from an Assessed Presentation 
during Class Time to a Video-based Assessment in a 
Spanish Culture Module 


Deborah Grange’’ 
Mari Paz Baliorea Enriquez”! 


Summary 


This case study focuses on an innovative approach to summative assessment on the ‘Studying the Hispanic, 


Luso-Brazilian and Native American Worlds’ module at level 4. 


Previously, students were required to make presentations during class time for a summative assessment. 
As the cohort consisted of about 50 students, they presented in groups rather than individually. This 
approach required a substantial percentage of limited face-to-face time, led to concerns about the fair 
allocation of individual marks, and demanded that students listen to group presentations of varying 


quality and sometimes limited value. 


The course tutor decided to substitute the in-class group presentation assignment with a requirement for 
students to submit videos individually, thus allowing for individual marks to be allocated and freeing 


up valuable class time. 


This video-based assessment took place during Autumn 2014, and required students to submit videos of 
no more than five minutes on topics provided by the course tutor. This case study describes the experience 
of students and staff. 


implementation 


Students were asked to record a five-minute video of a response to topics provided to them, and then upload 
it into Moodle’s assignment submission tool. The course tutor used some class time to briefly describe 
what was required and she reported that student reaction seemed positive. She also provided some 


links to advice on the technicalities via Moodle, which are listed in the Take Aways section below. 


The course tutor emphasised to students that she would not accept emailed assignments, and any 


students who emailed their video to her were told to submit it via the assignment tool. 


To offer support to students, the Learning Development Tutor held a drop-in surgery a few days before 
the assignment deadline, where students could use College equipment to record their videos. Only three 
students attended, and none of them wanted to record their video there and then. Instead, they wanted 


to check that they were using the appropriate approach with their smartphones or tablets. 


As Moodle’s assignment submission tool had an upload limit of 1OOMB, students were required to compress 


their videos before they submitted them. Most managed this process successfully without support. 


50 Case study author: Deborah Grange, Senior Learning Development Tutor (TEL), Centre for Transformative Practice in 
Learning and Teaching, Birkbeck, University of London. 


51 Project lead: Mari Paz Balibrea Enriquez, Senior Lecturer, Cultures & Languages, Birkbeck, University of London. 


89 


Evaluation 


All active students submitted this assignment, although six students required a substantial extension 


due to technical issues. 


Benefits and Challenges 
Benefits 


e Class time was reallocated and used more efficiently. 
e Individual assessment was enabled. 


e The course tutor was able to mark student work at a time convenient for her. 


Challenges 


Initial concerns that not all students had access to smartphones or other technology for videoing 
proved groundless. However, the requirement to compress the video necessitated by Moodle’s 1OOMB 
limit proved highly challenging for a minority of students, and substantial support was required for a 


few students. Problems included: 


e Two students submitted videos that had been truncated according to a default setting when 
emailing video from an iPhone. One of these students found a solution himself; the other 
required substantial support due to very weak IT skills (e.g. this student was unsure of how to 


upload an attachment to an email). 


e One student submitted a file in a format which could not be accessed by the course tutor, and 


simply re-submitted in a different format. 


e Inanother case, the student was not waiting long enough for the file to upload in Moodle, 


which was something that we had not considered explaining. 


Although comparatively few students required support, substantial staff time was required from TEL 
staff at the College. Also, where students had not submitted successfully, out-of-date email addresses 


made communication challenging. 


When the assignment was completed during class time, students would have benefited from observing 


particularly good presentations, but this was not possible when using online submission in Moodle. 


Take-Aways 


The upload limit of the standard online submission tool in Moodle made the task much more difficult 
for students. Use of the institution’s lecture capture software Panopto, for student-created content, is 
likely to resolve this problem. Furthermore, a shared folder on the Panopto server would enable 


students to see each other’s work selectively, thereby replicating the best of the in-class experience. 


Students were provided with the following links to help them compress their videos: 


e Advice for video compression using different types of phones and tablets: 
http://7labs.heypub.com/mobile/compress-video-android-ios-wp.html 

e Advice for video compression for recordings made with a video camera: 
http://blogs.it.ox.ac.uk/adamweblearn/2013/10/how-to-compress-large-video-files-without- 
losing-quality-using-handbrake/ 
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Chapter 23: Using Panopto for Formative 
Assessment in Earth Science 


Deborah Grange” 
Steve Hirons* 


Summary 


Students on the BSc Geology course at Birkbeck are required to produce descriptions of rocks so that, 
by the end of the course, they are able to describe rocks in detail. This practical aspect of the course is 
experiential, and continuous formative feedback throughout the course encourages and supports students 
in their development, and ensures they become competent with these descriptions. This is an essential 
outcome of the course, as 50% of the final exam is based on the student’s ability to write descriptions 
of rocks. Furthermore, it is important that they become confident at providing descriptions verbally 


because the degree culminates in a viva with a professional geologist. 


Previously, students would produce written descriptions and submit them for formative assessment. 
This had the disadvantage that only the marker and student would see the work and, obviously, this 
did not give students practice in talking about the rock. Therefore, the assessment was changed to enable 
students to record their verbal commentaries using Panopto (recording software licensed by the institution). 
These recordings were then uploaded to a shared Panopto folder so that all students on the module 
could access them. This gave students experience in providing verbal commentaries, and also allowed 


for them to see how their colleagues on the module had approached the task. 


Implementation 


The tutor, who had used Panopto previously, provided students with detailed instructions about both 
the task and the video platform, Panopto. Students downloaded the Panopto software onto their own 
computers. They then opened the images of rocks that the tutor had given them, added their verbal 
commentary, and then uploaded them to a single Panopto folder that served for all students on the 
module so that everybody could access, and learn from, the commentaries. The tutor and his colleagues 


provided formative feedback. 


As the task was for formative assessment, students were allowed to take as long as they liked to prepare. 
The tutor did not want to add pressure by applying a deadline, but encouraged a three-week turnaround. 
Generally, this preparation time took around two weeks. Many students reported feeling anxious and 
‘outside their comfort zone’ given the prospect of other students hearing their descriptions. Some 
dealt with that anxiety by writing down the entire script for the video, but then said that they would be 
more spontaneous next time. No particular time limit was given for the videos, and the longest one 


was about eight minutes. 


52 Case study author: Deborah Grange, Senior Learning Development Tutor (TEL), Centre for Transformative Practice in 
Learning and Teaching, Birkbeck, University of London. 


53 Project lead: Steve Hirons, Lecturer, Earth Sciences, Birkbeck, University of London. 
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Evaluation 


Overall feedback 


Overall, students reported that the assessment made them feel more confident and comfortable 


describing rocks. 
Distance learners reported more satisfaction than face-to-face. 


Students reported that they were more organised in writing their observations because they 


knew that they were going to be recorded. 


Students reflected on the difficulty of describing what they were seeing and said that they had 


made notes about what they would do differently next time. 
Some students said that they enjoyed making the recordings, despite their initial anxieties. 
Students reported that the formative nature of the assessment made it less stressful. 


Students said that they would feel more comfortable if they did it again. 


Individual comments 


One student didn't upload anything, but then saw what others had done and thought it was fantastic. 


Another student was very nervous and reluctant to participate, but then reported that carrying 


out the task has helped her gain confidence in public speaking. 


Another student who showed signs of struggling with English academic language reported 
that she would prefer to write down her answers. The public nature of the recording made it 


stressful for her. 


A very able student reported that it was extremely difficult and the most challenging task she 


had undertaken for a long time. It made her nervous. 


Another student reported that they found it very difficult because they had never done it before, 
but felt that they would enjoy it more the next time they did it, and that it would help with 
public speaking. 


One student said that it encouraged them to be more concise and to be more thoughtful about 


what they were saying. They also liked it because it was a ‘different way of learning’. 


Another student, who did not upload her own contribution, found it very valuable. She was 
able to recap without having asked basic questions which she would find embarrassing. She 
also thought that learning from different students who were at the same level as her helped 
her learn from a different angle. She reported feeling greater confidence and understanding 


about producing descriptions and will upload her own next time. 
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Benefits and Challenges 


Overall, students could see tangible benefits and it helped their confidence, although many reported 


feeling anxious about the assessment. 


For the tutor, the preparation for this assessment was extensive and meticulous, focusing on the desired 
learning outcomes. A detailed, step-by-step set of instructions was written for students, and headings 
were provided to help them structure their descriptions. Most students found the technicalities 


straightforward, although the step-by-step instructions provided by the project lead were very important. 


The Department of Earth and Planetary Sciences has distance learners as well as face-to-face, and one 


advantage of this initiative is that it is inclusive for distance learners. 


Take-Aways 


e Once the initial work has been done, the workload is manageable for the lecturer. 
e It may push students outside their comfort zones with the resultant discomfort and resistance. 


e The tutor in this case has always given lectures to students on how to organise and present their 
rock descriptions, but he thinks that the recording requirement makes students take it more 


seriously and take responsibility and ownership of their descriptions. 


e This exercise may also be useful for group work. For example, students could work in groups 


to generate the commentaries and/or give feedback on other groups’ work. 


e As well as providing the students with valuable experience, the tutor reports that it will inform 


his teaching practice, as it requires students to engage in reflection. 


e Students generally have a shrewd idea of academic ranking within the class, and the statistics 
(albeit very small at this stage) suggest that the contributions from the ‘top’ students attracted 


more views. 
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Chapter 24: Assessment and Feedback Video 


Sultan Wadud*‘ 
Vera Janev» 


Summary 


For the second SOAS Learning and Teaching Conference, which was themed around Innovative 
Approaches to Assessment and Feedback, we decided to produce a video entitled “SOAS Assessment 
and Feedback’. You can access the video here (https://youtu. be/YQORMH6d Yew). The video project 
involved interviewing both students and academics to capture experiences, tips, observations and 
recommendations. The purpose was to provide students with an insight into how they could utilise 


the feedback they receive for their assignments. 


The students were also asked what, with hindsight, they would have liked to have known when they 
had started their studies. 


Implementation 


We received training on how to use the HD cameras and Adobe Premiere Pro from the marketing team 
at SOAS. We also contacted the audiovisual team to obtain clip-on mics and lighting equipment to ensure 


that we captured the best possible quality recording. 


We contacted potential interviewees during the summer and at the beginning of term, and booked 30- 
minute slots to conduct the interviews. We were able to interview six students and six academics from 
a range of backgrounds and faculties. Where possible, we tried to conduct the interviews around the 
campus, i.e. in the Student Union, library or offices. In addition to the interviews, we recorded various 
‘beauty shots’ in parts of the campus, which were used in the final clip to complement the interviews 


and give viewers a sense of life at SOAS. 


Once all the recordings had been completed, the interviews were edited and reviewed; the parts that 


best suited the theme of the conference were combined to produce a five-minute clip. 


Evaluation 


The video was well received at the conference and was subsequently promoted within our study skills 


pages for students on the VLE. 


54 Case study author and project lead: Sultan Wadud, E-Learning Officer, SOAS, University of London. 


55 Project team member: Vera Janev, Learning and Teaching Development Officer, SOAS, University of London. 
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Benefits and Challenges 


Benefits 
e A short video appealed to students as the ‘talking heads’ were peers and experienced 
academics. 
e The full interviews contained content that is currently being edited to produce short clips for 
other Welcome Week activities. 
e It was useful to have a colleague who had experience in editing videos. 
Challenges 
e Due toa lack of available funds, existing staff were required to manage the whole video 
project. The main challenge was that we had to learn how to use the recording equipment and 
editing software. 
e Booking the interviewees was another challenge, especially during the summer when not 


many academics or students were around. 


Take-Aways 


Video projects can be very effective, but try and ensure that they are short and concise. 


Obtaining the right equipment and planning when to conduct the interviews can be difficult, 


so ensure you book this as soon as possible. 


Finally, editing the videos is more time-consuming than most people expect, so ensure 


sufficient time is planned for this before delivering the video project. 
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SECTION 6 


Case Studies — 


Technical Developments 


Chapter 25: SOAS Assignment Creation and Results 
Tracking 


Linda O’Sullivan** 
Andrew Leedham?” 


Summary 
Purpose 
The aim of this technical development was to reduce the administrative workload and risk of error by: 


e Automating the creation of Turnitin assignments. 
e Collecting the results from Moodle. 
e Automatically feeding into the student record management system. 


e As SOAS deducts marks for late submission on a per-day basis, the submission date also 


needed to be returned along with the grade. 


Activity summary 


1. Data is collected from the Turnitin tool, which includes grades and the submission date of 
each assignment submission, and is saved into a text file and passed to a sftp folder (a secured 


folder accessible online by more than one party). 


2. Once data is collected from Turnitin, it is then passed on to UNITe (student record management 
system). All data is initially written to a holding table and extracted from there to update 


individual records. 


The figure overleaf illustrates the requirements for the development and how data is passed between 


the various systems, which include UNITe, Moodle and Turnitin. 


56 Case study author and project lead: Linda O’Sullivan, E-Learning Manager, SOAS, University of London. 
57 Project support: Andrew Leedham, Senior Analyst Programmer, SOAS, University of London. 
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Student Record Management 
System (such as SITS, UNITe) 


1. Collect all the upcoming 


assessment details for an 
academic year and generate a 
CSV file containing those data 


2. CRON task: Deposit the CSV 
file to hosting server via SFTP 
connection 


7. CRON task: Read the CSV 
file via SFTP connection and 


update students record 


accordingly. 


Known issues 


Moodle 


Moodle Server / backend 


3. CRON task: Read each line from CSV file. Create 
TURNITIN instance in Moodle courses so student can 
submit their assessment 


6. CRON task: Collect assessment marks from Moodle 
database. Generate a CSV file and deposit this to the 
same location where it is accessible via SFTP connection 


Moodle site / frontend 


4. Students: Submit their assessment 


5. Teachers: Mark those submitted 
assessments 


This plugin was built for the Moodle plugin Turnitin Direct 1, so it is not guaranteed that it will work 


on version 2. 


All the default values for assignment creation are hard coded. The next version should have selectable options. 


Usage 


This plugin was developed for SOAS’s on-campus courses, but later on it was adopted by SOAS’s 


distance learning unit too. 


The plugin is owned by the University of London Computer Centre (ULCC). 
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Benefits and Challenges 
Benefits 


SOAS is heavily dependent on essays as an assessment method, so the automation of the assignment 
creation process has had a wide impact on the following areas: 
e Academic workload. The responsibility for creating assignments has been removed from 
individual academics. This had previously been a major stress point. 
e Administrative workload. The majority of assignments are managed automatically leaving 
only outliers to be managed manually. Errors have been greatly reduced. 
e VLE support workload. Support calls have been greatly reduced. 


e Quality Assurance. Due dates on assignments match those published on student materials. 


Challenges 


In order for the benefits to be fully realised, a consistent approach to assignments needs to be 
maintained. SOAS has addressed this by insisting that any changes to due dates, assignment numbers, 
etc., is managed by faculty administrative staff, ensuring that the data is consistent across all systems. 
The responsibility of supplying due dates still lies with academic staff, and extracting these in a timely 


manner so the assignments can be created in time has proved problematic in some instances. 


Dependence on Turnitin can be a challenge, and workarounds for common errors (e.g. failure to 


process reports) need to be worked out and consistently applied in consultation with all parties. 


Take-Aways 


While not perfect, this plugin has been a success. A conscious decision was made when designing it to 
not try to make it do all things for all circumstances. The aim was to automate the bulk processes, 


freeing time and resources to manually deal with outliers. 
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Chapter 26: My Feedback Report Plugin for Moodle 


Jessica Gramp* 


Summary 


The Moodle My Feedback report appears in the Profile > Reports menu and allows students to see an 
overview of all their grades and feedback for assessment activities, such as Moodle Assignments, Turnitin 
Assignments, workshops, quizzes and manual grade items entered directly into the GradeBook on one 
page. It also enables the information to be filtered, exported to Excel or printed. It provides students’ 
visible grades and a link to their submission, as well as any feedback that has been released to them. 
Anything hidden in the GradeBook or on the course will not display. Staff have access to dashboards 


of summary information about the students they support. 


The report is intended to help students, with the support of their personal tutors, understand the variety 
of feedback they receive. It can also be used to identify similarities between feedback received from 


across modules to help students see how they can improve their work in future assessments. 


Known issues 


The My Feedback report does not draw in feedback from Turnitin directly, as there is no API available to 
enable this. However, the report does allow students to copy and paste general feedback from Turnitin 
into the report, should they choose to do so. The report also makes it easier for students and staff to 
access Turnitin feedback, by providing links that open each assessment directly within Turnitin, so 
although the feedback is not automatically displayed within the report for comparison, it is still more 


easily available. 


Usage 


The My Feedback plugin is available for download from moodle.org and has been installed on over 300 


Moodle installations worldwide. 


University College London (UCL) piloted the My Feedback report with over 700 students from Engineering 
in 2015, and released it to all staff and students across the university in October 2016. Initial feedback 
from students and staff has been positive, and UCL is working to promote its use more widely and 


encourage staff to enter grades electronically within Moodle, so that these are available within the report. 


58 Case study author and project lead: Jessica Gramp, Digital Education Advisor, UCL. 
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Chapter 27: Coursework Module for Moodle 


Sarah Sherman?’ 
Alistair Spark’? 


Summary 


The purpose of the Coursework module, specifically developed in the context of the UK higher education 


market, is to improve Moodle’s function for summative coursework assignments. 


This plugin, which is an alternative to the Moodle Assignment tool, supports a variety of marking workflows 
that are not supported by other tools. It allows for double-blind marking, sample marking and single 
marking with moderation, which are the three key workflows for marking practice in the UK HE sector 
and many more. The allocation of students or groups to specific markers can be arranged either manually 
or automatically. It integrates with the Plagiarism API (for Turnitin) and the Moodle Advanced Grading 
API (to make use of Moodle Scales and Rubrics). It also supports assignments with personal deadlines, 
no deadlines and the regular ones where everyone has the same deadline. In addition, it has the ability 


to provide individual students with an extension. 


Coursework has been piloted and subsequently rolled out at the RVC, which developed it in partnership 
with the University of London Computer Centre (ULCC), the University of Plymouth and LSHTM. It 
was a major evolution of the earlier Jisc-funded project, managed by the University of Exeter, called 


OCM (Online Coursework Management). 


Known issues 


The tool is still undergoing regular updates, but the core functions are now complete. During the pilot, 
student submission of assignments was flawless; there were no reported technical issues from staff or 
students. The double-blind marking workflow worked well; however, markers involved in the pilot 
were notably late in marking their assignments and well beyond their set deadlines for initial marking. 
Although not a flaw with the tool itself, since this was purely due to academic staff time pressures, this 


is likely to remain a trending concern during any rollout. 


Coursework does also require a fair amount of Moodle administration time in order to match the 


institution’s particular marking workflow variants, but that is also what makes it widely usable. 


59 Case study author: Sarah Sherman, Service Manager, Bloomsbury Learning Environment. 


60 Project lead: Alistair Spark, VLE Manager, RVC, University of London. 
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Usage 


Coursework was piloted at the RVC in Spring and Summer 2015 across three courses, each with different 
aims, grading scales and marking workflows. A number of issues and improvements were identified 
during these pilots beyond the previously identified requirements for a full rollout at scale. The pilots 
were run with a Minimum Viable Product (MVP) in order to tease out any issues with the permissions 
of the different roles, assess the willingness of markers to move to online marking and feedback, and 


identify any further functionality requirements. 


Coursework was subsequently rolled out fully at the RVC throughout the 2016-17 academic year, 
following the presentation of a paper presented to the Learning, Teaching and Assessment Committee 
in May 2016. Its use remains optional, but is the recommended solution for any paperless submission 
of summative work and, as of June 2017, is currently used for more than 90% of submissions. Rollout 
for distance learning courses began in Spring 2017 following the availability of a ‘no deadline’ 
assignment type. The only holdouts are for assignments where a single marker is marking over 60 


papers, a situation in which the markers have a preference for paper. 


Benefits and Challenges 


Following a consultation exercise, an overwhelming majority of markers were confident about marking 
online and giving feedback, with 88% of surveyed respondents being either ‘very confident’ or ‘confident’ 
about using the tool. While this was very positive, markers felt much less confident about PDF annotation, 
therefore a PDF annotation toolkit was developed with information on tools available on the various 


platforms and relevant existing online guidance available. 


All markers who responded to the post-pilot survey stated that they would be comfortable marking all 
future submissions through Coursework. Feelings were mixed, however, about rolling this out to all 
submissions. Overall, it is really positive that markers are so open to online marking and feedback. 
Indeed, if they did not already have some prior experience of electronic assessment, this was solely due 
to lack of opportunity not willingness. More than 75% of markers expected that Coursework would be 


beneficial to them. 


Time is, however, a major concern for markers, in particular due to the absence of a physical pile of 
paper on a desk to remind them that they must mark the papers, and the reduced visibility for managers 


of the marking workload. 


One of the concerns before the pilot was to ensure that markers would review the Turnitin similarity 
reports of student submissions; from the pre-pilot questionnaire, 74% of respondents reported that 


they would either review all of the submissions or specific papers if they suspected plagiarism. 


Take-Aways 


Staff training was adapted based on the pre-pilot survey, with online guides and video walkthroughs 
for markers. Some on-site workshops were scheduled, which became feedback sessions rather than 


training sessions since the markers had already finished marking successfully. 
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